[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:36:59 dist-test-slave-w6c0 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:36:59 dist-test-slave-w6c0 krb5kdc[395](info): set up 2 sockets
Sep 28 08:36:59 dist-test-slave-w6c0 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 08:37:00.767776 388 mini_kdc.cc:121] Time spent starting KDC: real 2.126s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:37:00 dist-test-slave-w6c0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:37:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:37:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:37:00.961275 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39659
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:01.296262 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:01.296634 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:01.296808 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:01.304994 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:37:01.305179 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:01.305294 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:01.305415 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:37:01.305503 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:37:01.316550 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:01.317987 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:01.319408 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:01.332243 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:01.332325 408 server_base.cc:1034] running on GCE node
W20260928 08:37:01.332156 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:01.332412 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:01.333429 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:01.334972 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:01.336176 408 hybrid_clock.cc:648] HybridClock initialized: now 1790584621336102 us; error 61 us; skew 500 ppm
I20260928 08:37:01.336547 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:01 dist-test-slave-w6c0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:01.339706 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:37:01.341212 408 webserver.cc:460] Webserver started at http://127.0.97.62:39927/ using document root <none> and password file <none>
I20260928 08:37:01.341915 408 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:01.342041 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:01.342360 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:01.345340 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "b57755b5f0454334b6f52c4ef0864392"
format_stamp: "Formatted at 2026-09-28 08:37:01 on dist-test-slave-w6c0"
server_key: "8b525ca00fb19bd2bb164ed9dac86633"
server_key_iv: "9906253ea105c3e942327ef7b087411c"
server_key_version: "encryptionkey@0"
I20260928 08:37:01.346109 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "b57755b5f0454334b6f52c4ef0864392"
format_stamp: "Formatted at 2026-09-28 08:37:01 on dist-test-slave-w6c0"
server_key: "8b525ca00fb19bd2bb164ed9dac86633"
server_key_iv: "9906253ea105c3e942327ef7b087411c"
server_key_version: "encryptionkey@0"
I20260928 08:37:01.352065 408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260928 08:37:01.355665 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:01.357896 408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 08:37:01.358119 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "b57755b5f0454334b6f52c4ef0864392"
format_stamp: "Formatted at 2026-09-28 08:37:01 on dist-test-slave-w6c0"
server_key: "8b525ca00fb19bd2bb164ed9dac86633"
server_key_iv: "9906253ea105c3e942327ef7b087411c"
server_key_version: "encryptionkey@0"
I20260928 08:37:01.358340 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:01.380797 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:01.385216 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:01.385522 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:01.406041 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39659
I20260928 08:37:01.406062 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39659 every 8 connection(s)
I20260928 08:37:01.407495 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:37:01.408603 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 408
I20260928 08:37:01.408950 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:37:01.410924 388 external_mini_cluster.cc:1428] Setting key a178768a259bb1f8913c64f3f0e24c19
I20260928 08:37:01.411993 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:01.423681 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392: Bootstrap starting.
I20260928 08:37:01.427040 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:01.428093 475 log.cc:824] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:01.431743 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392: No bootstrap required, opened a new log
Sep 28 08:37:01 dist-test-slave-w6c0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:01.439649 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57755b5f0454334b6f52c4ef0864392" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39659 } }
I20260928 08:37:01.440224 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:01.440362 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57755b5f0454334b6f52c4ef0864392, State: Initialized, Role: FOLLOWER
I20260928 08:37:01.441035 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57755b5f0454334b6f52c4ef0864392" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39659 } }
I20260928 08:37:01.441314 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:37:01.441461 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:37:01.441679 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:01.444661 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57755b5f0454334b6f52c4ef0864392" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39659 } }
I20260928 08:37:01.445202 475 leader_election.cc:302] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b57755b5f0454334b6f52c4ef0864392; no voters:
I20260928 08:37:01.445870 475 leader_election.cc:288] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:37:01.445979 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:01.447332 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392 [term 1 LEADER]: Becoming Leader. State: Replica: b57755b5f0454334b6f52c4ef0864392, State: Running, Role: LEADER
I20260928 08:37:01.447923 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57755b5f0454334b6f52c4ef0864392" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39659 } }
I20260928 08:37:01.448488 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:37:01.450289 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:01.414865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47132 (local address 127.0.97.62:39659)
0928 08:37:01.415651 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:01.415662 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:01.416838 (+ 1176us) server_negotiation.cc:407] Connection header received
0928 08:37:01.425857 (+ 9019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:01.425879 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:01.426279 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:01.426731 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:01.430082 (+ 3351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:01.431528 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:01.433308 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:01.434082 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:01.434334 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:01.439889 (+ 5555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:01.439901 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:01.439916 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:01.440260 (+ 344us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:01.444236 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:01.445164 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:01.445181 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:01.445187 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:01.445433 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:01.445898 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:01.445905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:01.445909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:01.446948 (+ 1039us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:01.447835 (+ 887us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:01.448463 (+ 628us) server_negotiation.cc:299] Negotiation successful
0928 08:37:01.448965 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":367,"threads_started":1}
I20260928 08:37:01.453719 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b57755b5f0454334b6f52c4ef0864392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57755b5f0454334b6f52c4ef0864392" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39659 } } }
I20260928 08:37:01.454178 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:01.454802 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b57755b5f0454334b6f52c4ef0864392. Latest consensus state: current_term: 1 leader_uuid: "b57755b5f0454334b6f52c4ef0864392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57755b5f0454334b6f52c4ef0864392" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39659 } } }
I20260928 08:37:01.455137 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:01.457198 487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:37:01.463025 487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:37:01.472724 487 catalog_manager.cc:1347] Generated new cluster ID: d206fc506d1f4871bed578877bc7991a
I20260928 08:37:01.472843 487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:37:01.490458 487 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:37:01.491881 487 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:37:01.504906 487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b57755b5f0454334b6f52c4ef0864392: Generated new TSK 0
I20260928 08:37:01.505856 487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:37:01.558706 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39659
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:01.911728 502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:01.912036 502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:01.912210 502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:01.920168 502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:01.920297 502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:01.920488 502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:37:01.930747 502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39659
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:01.932116 502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:01.933521 502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:01.947937 508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:01.947922 507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:01.947958 510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:01.949085 502 server_base.cc:1034] running on GCE node
I20260928 08:37:01.949901 502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:01.950986 502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:01.952220 502 hybrid_clock.cc:648] HybridClock initialized: now 1790584621952183 us; error 27 us; skew 500 ppm
I20260928 08:37:01.952553 502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:01 dist-test-slave-w6c0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:01.955984 502 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:37:01.957602 502 webserver.cc:460] Webserver started at http://127.0.97.1:41003/ using document root <none> and password file <none>
I20260928 08:37:01.958326 502 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:01.958572 502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:01.958899 502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:01.961901 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "f68f352f29c34ce7bcd6a346fd5d5f0b"
format_stamp: "Formatted at 2026-09-28 08:37:01 on dist-test-slave-w6c0"
server_key: "74408fb49d363f222611bb616b61416b"
server_key_iv: "52d53fb15ec87c332b456a84a297ebb7"
server_key_version: "encryptionkey@0"
I20260928 08:37:01.962859 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f68f352f29c34ce7bcd6a346fd5d5f0b"
format_stamp: "Formatted at 2026-09-28 08:37:01 on dist-test-slave-w6c0"
server_key: "74408fb49d363f222611bb616b61416b"
server_key_iv: "52d53fb15ec87c332b456a84a297ebb7"
server_key_version: "encryptionkey@0"
I20260928 08:37:01.968816 502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260928 08:37:01.973080 517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:01.975128 502 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 08:37:01.975380 502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "f68f352f29c34ce7bcd6a346fd5d5f0b"
format_stamp: "Formatted at 2026-09-28 08:37:01 on dist-test-slave-w6c0"
server_key: "74408fb49d363f222611bb616b61416b"
server_key_iv: "52d53fb15ec87c332b456a84a297ebb7"
server_key_version: "encryptionkey@0"
I20260928 08:37:01.975611 502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:01.991959 502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:01.996483 502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:01.996793 502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:01.997836 502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:01.999251 502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:01.999373 502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:01.999526 502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:01.999606 502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:02.040872 502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41383
I20260928 08:37:02.040908 629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41383 every 8 connection(s)
I20260928 08:37:02.042344 502 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:37:02.048302 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 502
I20260928 08:37:02.048543 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:37:02.049211 388 external_mini_cluster.cc:1428] Setting key 5e6aa59eb71c15080c3b914b414b6b41
Sep 28 08:37:02 dist-test-slave-w6c0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:02.063531 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:02.045475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52987 (local address 127.0.97.62:39659)
0928 08:37:02.045934 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:02.045939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:02.047078 (+ 1139us) server_negotiation.cc:407] Connection header received
0928 08:37:02.049121 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:02.049128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:02.049228 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:02.049361 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:02.051180 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:02.051944 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:02.053581 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:02.053999 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:02.054140 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:02.057762 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:02.057767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:02.057771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:02.057802 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:02.060410 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:02.061142 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:02.061152 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:02.061154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:02.061265 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:02.061703 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:02.061708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:02.061711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:02.061982 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:02.062178 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:02.062943 (+ 765us) server_negotiation.cc:299] Negotiation successful
0928 08:37:02.063164 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":261,"threads_started":1}
I20260928 08:37:02.064551 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:02.046139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39659 (local address 127.0.97.1:52987)
0928 08:37:02.046914 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:02.046958 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:02.048800 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:02.049598 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:02.049611 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:02.050299 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:02.051004 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:02.051031 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:02.052105 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:02.052110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:02.053436 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:02.053448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:02.054245 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:02.054254 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:02.054494 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:02.055303 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:02.055325 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:02.057505 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:02.060565 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:02.060575 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:02.060594 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:02.061011 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:02.061377 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:02.061382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:02.061386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:02.061577 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:02.062110 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:02.062119 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:02.062559 (+ 440us) client_negotiation.cc:769] Sending connection context
0928 08:37:02.062832 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 08:37:02.063400 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":348,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:37:02.066222 630 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39659
I20260928 08:37:02.066613 630 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:02.067413 630 heartbeater.cc:511] Master 127.0.97.62:39659 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 08:37:02.069937 440 ts_manager.cc:194] Registered new tserver with Master: f68f352f29c34ce7bcd6a346fd5d5f0b (127.0.97.1:41383)
I20260928 08:37:02.072630 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:37:02.091993 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39659
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:02.429631 636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:02.429992 636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:02.430183 636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:02.438267 636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:02.438380 636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:02.438612 636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:37:02.448587 636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39659
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:02.449962 636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:02.451354 636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:02.464653 641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:02.464708 642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:02.464807 644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:02.465387 636 server_base.cc:1034] running on GCE node
I20260928 08:37:02.466042 636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:02.467036 636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:02.468261 636 hybrid_clock.cc:648] HybridClock initialized: now 1790584622468217 us; error 57 us; skew 500 ppm
I20260928 08:37:02.468568 636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:02 dist-test-slave-w6c0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:02.472200 636 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:37:02.473596 636 webserver.cc:460] Webserver started at http://127.0.97.2:44173/ using document root <none> and password file <none>
I20260928 08:37:02.474289 636 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:02.474427 636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:02.474748 636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:02.477669 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "ceed64e3985b4c81a2872a45614bc51c"
format_stamp: "Formatted at 2026-09-28 08:37:02 on dist-test-slave-w6c0"
server_key: "ebedca20b8e9ffbbcc088565358f8584"
server_key_iv: "c111781c73e334c3eb910e493da8ca15"
server_key_version: "encryptionkey@0"
I20260928 08:37:02.478437 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ceed64e3985b4c81a2872a45614bc51c"
format_stamp: "Formatted at 2026-09-28 08:37:02 on dist-test-slave-w6c0"
server_key: "ebedca20b8e9ffbbcc088565358f8584"
server_key_iv: "c111781c73e334c3eb910e493da8ca15"
server_key_version: "encryptionkey@0"
I20260928 08:37:02.483636 636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 08:37:02.486949 651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:02.488240 636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:37:02.488409 636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "ceed64e3985b4c81a2872a45614bc51c"
format_stamp: "Formatted at 2026-09-28 08:37:02 on dist-test-slave-w6c0"
server_key: "ebedca20b8e9ffbbcc088565358f8584"
server_key_iv: "c111781c73e334c3eb910e493da8ca15"
server_key_version: "encryptionkey@0"
I20260928 08:37:02.488600 636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:02.511125 636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:02.515101 636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:02.515389 636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:02.516341 636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:02.517725 636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:02.517817 636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:02.517982 636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:02.518031 636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:02.554199 636 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42743
I20260928 08:37:02.554246 763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42743 every 8 connection(s)
I20260928 08:37:02.555500 636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:37:02.558786 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 636
I20260928 08:37:02.559015 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:37:02.559444 388 external_mini_cluster.cc:1428] Setting key c1c7e00a92c3d591e622af4f1fa5afae
Sep 28 08:37:02 dist-test-slave-w6c0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 08:37:02.576063 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:02.558476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48917 (local address 127.0.97.62:39659)
0928 08:37:02.558684 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:02.558690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:02.560592 (+ 1902us) server_negotiation.cc:407] Connection header received
0928 08:37:02.562419 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:02.562424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:02.562522 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:02.562628 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:02.564085 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:02.564801 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:02.566529 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:02.566958 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:02.567094 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:02.570426 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:02.570434 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:02.570443 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:02.570473 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:02.573040 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:02.573740 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:02.573745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:02.573749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:02.573874 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:02.574295 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:02.574301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:02.574305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:02.574667 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:02.574816 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:02.575485 (+ 669us) server_negotiation.cc:299] Negotiation successful
0928 08:37:02.575715 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 08:37:02.577052 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:02.559210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39659 (local address 127.0.97.2:48917)
0928 08:37:02.560430 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:02.560481 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:02.562170 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:02.562904 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:02.562912 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:02.563402 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:02.563927 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:02.563943 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:02.564964 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:02.564968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:02.566358 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:02.566370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:02.567204 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:02.567213 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:02.567421 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:02.568118 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:02.568144 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:02.570214 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:02.573178 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:02.573187 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:02.573202 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:02.573614 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:02.573992 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:02.573997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:02.574001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:02.574183 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:02.574801 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:02.574809 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:02.575138 (+ 329us) client_negotiation.cc:769] Sending connection context
0928 08:37:02.575400 (+ 262us) client_negotiation.cc:240] Negotiation successful
0928 08:37:02.575964 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"thread_start_us":778,"threads_started":1}
I20260928 08:37:02.578532 764 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39659
I20260928 08:37:02.578853 764 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:02.579643 764 heartbeater.cc:511] Master 127.0.97.62:39659 requested a full tablet report, sending...
I20260928 08:37:02.581118 440 ts_manager.cc:194] Registered new tserver with Master: ceed64e3985b4c81a2872a45614bc51c (127.0.97.2:42743)
I20260928 08:37:02.581894 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:37:02.598568 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39659
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:02.937685 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:02.937963 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:02.938130 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:02.945158 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:02.945295 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:02.945477 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:37:02.955755 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39659
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:02.957108 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:02.958493 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:02.972347 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:02.972273 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:02.972687 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:02.973426 769 server_base.cc:1034] running on GCE node
I20260928 08:37:02.973994 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:02.975008 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:02.976250 769 hybrid_clock.cc:648] HybridClock initialized: now 1790584622976208 us; error 37 us; skew 500 ppm
I20260928 08:37:02.976555 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:02 dist-test-slave-w6c0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:02.979686 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:37:02.981084 769 webserver.cc:460] Webserver started at http://127.0.97.3:45241/ using document root <none> and password file <none>
I20260928 08:37:02.981736 769 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:02.981830 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:02.982074 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:02.985067 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "38ae980628ad43729d2cc70716b1918a"
format_stamp: "Formatted at 2026-09-28 08:37:02 on dist-test-slave-w6c0"
server_key: "5034962e3150c38b72e41e173f9ae1b9"
server_key_iv: "dbd151c8a13c475754219e058086578f"
server_key_version: "encryptionkey@0"
I20260928 08:37:02.985805 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "38ae980628ad43729d2cc70716b1918a"
format_stamp: "Formatted at 2026-09-28 08:37:02 on dist-test-slave-w6c0"
server_key: "5034962e3150c38b72e41e173f9ae1b9"
server_key_iv: "dbd151c8a13c475754219e058086578f"
server_key_version: "encryptionkey@0"
I20260928 08:37:02.990962 769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 08:37:02.994372 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:02.995659 769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:37:02.995820 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "38ae980628ad43729d2cc70716b1918a"
format_stamp: "Formatted at 2026-09-28 08:37:02 on dist-test-slave-w6c0"
server_key: "5034962e3150c38b72e41e173f9ae1b9"
server_key_iv: "dbd151c8a13c475754219e058086578f"
server_key_version: "encryptionkey@0"
I20260928 08:37:02.996021 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:03.014022 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:03.018283 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:03.018615 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:03.019570 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:03.020897 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:03.020969 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:03.021150 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:03.021207 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:03.060046 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37457
I20260928 08:37:03.060087 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37457 every 8 connection(s)
I20260928 08:37:03.061275 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:37:03.066156 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 769
I20260928 08:37:03.066331 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:37:03.066816 388 external_mini_cluster.cc:1428] Setting key 7a1ebc041b7ae9a158ce343d15b0cb93
Sep 28 08:37:03 dist-test-slave-w6c0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:03.075789 630 heartbeater.cc:503] Master 127.0.97.62:39659 was elected leader, sending a full tablet report...
I20260928 08:37:03.080981 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:03.064236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51093 (local address 127.0.97.62:39659)
0928 08:37:03.064418 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:03.064422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:03.065514 (+ 1092us) server_negotiation.cc:407] Connection header received
0928 08:37:03.067517 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:03.067524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:03.067610 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:03.067723 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:03.069339 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:03.070209 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:03.071757 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:03.072137 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:03.072230 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:03.075330 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:03.075335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:03.075338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:03.075365 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:03.078213 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:03.078897 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:03.078900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:03.078903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:03.079018 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:03.079376 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:03.079379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:03.079381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:03.079631 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:03.079739 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:03.080407 (+ 668us) server_negotiation.cc:299] Negotiation successful
0928 08:37:03.080577 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 08:37:03.082023 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:03.064730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39659 (local address 127.0.97.3:51093)
0928 08:37:03.065361 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:03.065400 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:03.067237 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:03.067982 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:03.067993 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:03.068568 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:03.069162 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:03.069189 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:03.070364 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:03.070369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:03.071630 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:03.071642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:03.072313 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:03.072317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:03.072496 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:03.073087 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:03.073111 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:03.075145 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:03.078325 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:03.078331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:03.078341 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:03.078793 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:03.079107 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:03.079110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:03.079112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:03.079281 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:03.079786 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:03.079791 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:03.080066 (+ 275us) client_negotiation.cc:769] Sending connection context
0928 08:37:03.080303 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 08:37:03.080713 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":246,"threads_started":1}
I20260928 08:37:03.083653 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39659
I20260928 08:37:03.083945 897 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:03.084635 897 heartbeater.cc:511] Master 127.0.97.62:39659 requested a full tablet report, sending...
I20260928 08:37:03.085906 440 ts_manager.cc:194] Registered new tserver with Master: 38ae980628ad43729d2cc70716b1918a (127.0.97.3:37457)
I20260928 08:37:03.086750 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51093
I20260928 08:37:03.094802 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:37:03 dist-test-slave-w6c0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584623, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:37:03 dist-test-slave-w6c0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584623, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:37:03.139050 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:03.123841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.0.97.1:41383)
0928 08:37:03.124444 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:03.124450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:03.124479 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:37:03.124544 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:03.124550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:03.124777 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:03.124909 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:03.126260 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:03.127348 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:03.128588 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:03.129535 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:03.129631 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:03.133557 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:03.133567 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:03.133582 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:03.133619 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:03.136546 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:03.137164 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:03.137172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:03.137176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:03.137294 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:03.137616 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:03.137619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:03.137621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:03.137954 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:03.138174 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:03.138424 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 08:37:03.138698 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":309,"threads_started":1}
W20260928 08:37:03.140368 545 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43396
Sep 28 08:37:03 dist-test-slave-w6c0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:37:03 dist-test-slave-w6c0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:37:03.182868 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:03.169389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.0.97.1:41383)
0928 08:37:03.169638 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:03.169657 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:03.169871 (+ 214us) server_negotiation.cc:407] Connection header received
0928 08:37:03.169995 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:03.169998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:03.170062 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:03.170149 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:03.171418 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:03.172100 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:03.173268 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:03.173655 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:03.173758 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:03.177419 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:03.177424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:03.177428 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:03.177459 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:03.180467 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:03.181122 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:03.181126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:03.181128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:03.181275 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:03.181612 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:03.181616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:03.181618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:03.181870 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:03.182042 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:03.182300 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 08:37:03.182452 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 08:37:03.184333 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 502
I20260928 08:37:03.197378 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:03.185325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.0.97.1:41383)
0928 08:37:03.185521 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:03.185526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:03.185732 (+ 206us) server_negotiation.cc:407] Connection header received
0928 08:37:03.185907 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:03.185912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:03.185972 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:03.186068 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:03.187240 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:03.187920 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:03.189159 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:03.189555 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:03.189700 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:03.191963 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:03.191968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:03.191973 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:03.192008 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:03.194915 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:03.195525 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:03.195531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:03.195535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:03.195668 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:03.196024 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:03.196029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:03.196032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:03.196279 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:03.196515 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:03.196815 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 08:37:03.197018 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:37:03.198294 625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:03.283344 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 502
I20260928 08:37:03.299746 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 636
Sep 28 08:37:03 dist-test-slave-w6c0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:37:03.315068 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:03.300810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45738 (local address 127.0.97.2:42743)
0928 08:37:03.301350 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:03.301358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:03.301383 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:37:03.301447 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:03.301454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:03.301662 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:03.301793 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:03.302971 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:03.303986 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:03.305146 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:03.305708 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:03.305803 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:03.309789 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:03.309798 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:03.309812 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:03.309844 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:03.312540 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:03.313162 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:03.313171 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:03.313175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:03.313313 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:03.313657 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:03.313660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:03.313662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:03.313972 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:03.314192 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:03.314587 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 08:37:03.314790 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":286,"threads_started":1}
I20260928 08:37:03.316543 759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:03.399801 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 636
I20260928 08:37:03.422760 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 769
Sep 28 08:37:03 dist-test-slave-w6c0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:37:03.438023 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:03.423807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48932 (local address 127.0.97.3:37457)
0928 08:37:03.424396 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:03.424402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:03.424430 (+ 28us) server_negotiation.cc:407] Connection header received
0928 08:37:03.424490 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:03.424496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:03.424710 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:03.424837 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:03.426049 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:03.427163 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:03.428409 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:03.428954 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:03.429050 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:03.433031 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:03.433043 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:03.433066 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:03.433101 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:03.435722 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:03.436271 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:03.436277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:03.436279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:03.436400 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:03.436710 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:03.436713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:03.436715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:03.437018 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:03.437230 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:03.437459 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 08:37:03.437642 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":316,"threads_started":1}
I20260928 08:37:03.439308 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:03.519908 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 769
I20260928 08:37:03.535969 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 408
I20260928 08:37:03.537144 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:03.638789 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 408
2026-09-28T08:37:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5049 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:37:03 dist-test-slave-w6c0 krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:37:03 dist-test-slave-w6c0 krb5kdc[918](info): set up 2 sockets
Sep 28 08:37:03 dist-test-slave-w6c0 krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260928 08:37:04.702611 388 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:37:04 dist-test-slave-w6c0 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:37:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:37:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:37:04.902657 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43087
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:05.216346 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:05.216673 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:05.216826 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:05.223773 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:37:05.223914 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:05.224036 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:05.224133 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:37:05.224223 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:37:05.233718 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:05.235052 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:05.236339 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:05.247786 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:05.247800 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:05.247826 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:05.248607 931 server_base.cc:1034] running on GCE node
I20260928 08:37:05.249311 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:05.250651 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:05.251863 931 hybrid_clock.cc:648] HybridClock initialized: now 1790584625251825 us; error 29 us; skew 500 ppm
I20260928 08:37:05.252305 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:05 dist-test-slave-w6c0 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:05.261839 931 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:37:05.263559 931 webserver.cc:460] Webserver started at http://127.0.97.62:37855/ using document root <none> and password file <none>
I20260928 08:37:05.264302 931 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:05.264444 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:05.264794 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:05.268035 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "3d5d4c3dc21c474a8c51e8fcdcfb316e"
format_stamp: "Formatted at 2026-09-28 08:37:05 on dist-test-slave-w6c0"
server_key: "9eb3a283bf81b93440e822ba65425be9"
server_key_iv: "0e05b17b8ea4fe99337f805f8da2e995"
server_key_version: "encryptionkey@0"
I20260928 08:37:05.268841 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "3d5d4c3dc21c474a8c51e8fcdcfb316e"
format_stamp: "Formatted at 2026-09-28 08:37:05 on dist-test-slave-w6c0"
server_key: "9eb3a283bf81b93440e822ba65425be9"
server_key_iv: "0e05b17b8ea4fe99337f805f8da2e995"
server_key_version: "encryptionkey@0"
I20260928 08:37:05.274147 931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:37:05.277607 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:05.279011 931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 08:37:05.279201 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "3d5d4c3dc21c474a8c51e8fcdcfb316e"
format_stamp: "Formatted at 2026-09-28 08:37:05 on dist-test-slave-w6c0"
server_key: "9eb3a283bf81b93440e822ba65425be9"
server_key_iv: "0e05b17b8ea4fe99337f805f8da2e995"
server_key_version: "encryptionkey@0"
I20260928 08:37:05.279451 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:05.295967 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:05.300189 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:05.300498 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:05.319876 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43087
I20260928 08:37:05.319897 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43087 every 8 connection(s)
I20260928 08:37:05.321303 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:37:05.325021 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:05.328526 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 931
I20260928 08:37:05.328753 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:37:05.329560 388 external_mini_cluster.cc:1428] Setting key b49988a995ab931e6ac208904f6871c3
I20260928 08:37:05.338142 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e: Bootstrap starting.
I20260928 08:37:05.341799 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:05.343124 998 log.cc:824] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e: Log is configured to *not* fsync() on all Append() calls
Sep 28 08:37:05 dist-test-slave-w6c0 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:05.346037 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e: No bootstrap required, opened a new log
I20260928 08:37:05.352239 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:05.331663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.0.97.62:43087)
0928 08:37:05.332399 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:05.332410 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:05.332450 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:37:05.334028 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:05.334048 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:05.334412 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:05.334811 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:05.336541 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:05.337797 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:05.339520 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:05.340167 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:05.340299 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:05.345192 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:05.345198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:05.345212 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:05.345246 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:05.348398 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:05.349153 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:05.349159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:05.349164 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:05.349326 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:05.349699 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:05.349702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:05.349704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:05.350186 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:05.350404 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:05.350914 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 08:37:05.351260 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":356,"threads_started":1}
I20260928 08:37:05.355005 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5d4c3dc21c474a8c51e8fcdcfb316e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43087 } }
I20260928 08:37:05.355391 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:05.355499 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5d4c3dc21c474a8c51e8fcdcfb316e, State: Initialized, Role: FOLLOWER
I20260928 08:37:05.356086 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5d4c3dc21c474a8c51e8fcdcfb316e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43087 } }
I20260928 08:37:05.356319 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:37:05.356477 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:37:05.356649 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:05.358729 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5d4c3dc21c474a8c51e8fcdcfb316e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43087 } }
I20260928 08:37:05.359200 998 leader_election.cc:302] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d5d4c3dc21c474a8c51e8fcdcfb316e; no voters:
I20260928 08:37:05.359786 998 leader_election.cc:288] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:37:05.359875 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:05.361501 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e [term 1 LEADER]: Becoming Leader. State: Replica: 3d5d4c3dc21c474a8c51e8fcdcfb316e, State: Running, Role: LEADER
I20260928 08:37:05.361979 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:37:05.362035 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5d4c3dc21c474a8c51e8fcdcfb316e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43087 } }
I20260928 08:37:05.365420 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d5d4c3dc21c474a8c51e8fcdcfb316e. Latest consensus state: current_term: 1 leader_uuid: "3d5d4c3dc21c474a8c51e8fcdcfb316e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5d4c3dc21c474a8c51e8fcdcfb316e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43087 } } }
I20260928 08:37:05.365767 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:05.366227 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d5d4c3dc21c474a8c51e8fcdcfb316e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5d4c3dc21c474a8c51e8fcdcfb316e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43087 } } }
I20260928 08:37:05.366525 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:05.370814 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:37:05.375833 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:37:05.384626 1010 catalog_manager.cc:1347] Generated new cluster ID: 5adb658312234357a6d9492ab09948b2
I20260928 08:37:05.384765 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:37:05.392503 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:37:05.394166 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:37:05.410938 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d5d4c3dc21c474a8c51e8fcdcfb316e: Generated new TSK 0
I20260928 08:37:05.411897 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:37:05.458587 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43087
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:05.776387 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:05.776672 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:05.776837 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:05.784322 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:05.784488 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:05.784658 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:37:05.794190 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43087
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:05.795537 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:05.796886 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:05.809655 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:05.809628 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:05.809588 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:05.810456 1024 server_base.cc:1034] running on GCE node
I20260928 08:37:05.811041 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:05.811901 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:05.813117 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790584625812970 us; error 137 us; skew 500 ppm
I20260928 08:37:05.813446 1024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:05 dist-test-slave-w6c0 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:05.816664 1024 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:37:05.818030 1024 webserver.cc:460] Webserver started at http://127.0.97.1:35611/ using document root <none> and password file <none>
I20260928 08:37:05.818737 1024 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:05.818840 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:05.819089 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:05.821828 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "a1148a7146054140869bd9f706eb33de"
format_stamp: "Formatted at 2026-09-28 08:37:05 on dist-test-slave-w6c0"
server_key: "0116e74639fba938dcf95be86d142bcb"
server_key_iv: "34074ab64a14d375d59fd80d51f66328"
server_key_version: "encryptionkey@0"
I20260928 08:37:05.822603 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a1148a7146054140869bd9f706eb33de"
format_stamp: "Formatted at 2026-09-28 08:37:05 on dist-test-slave-w6c0"
server_key: "0116e74639fba938dcf95be86d142bcb"
server_key_iv: "34074ab64a14d375d59fd80d51f66328"
server_key_version: "encryptionkey@0"
I20260928 08:37:05.827473 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 08:37:05.830693 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:05.831961 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:37:05.832123 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "a1148a7146054140869bd9f706eb33de"
format_stamp: "Formatted at 2026-09-28 08:37:05 on dist-test-slave-w6c0"
server_key: "0116e74639fba938dcf95be86d142bcb"
server_key_iv: "34074ab64a14d375d59fd80d51f66328"
server_key_version: "encryptionkey@0"
I20260928 08:37:05.832319 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:05.848099 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:05.852360 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:05.852671 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:05.853600 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:05.854961 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:05.855062 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:05.855213 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:05.855285 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:05.889092 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39927
I20260928 08:37:05.889132 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39927 every 8 connection(s)
I20260928 08:37:05.890391 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:37:05.895896 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 1024
I20260928 08:37:05.896123 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:37:05.896720 388 external_mini_cluster.cc:1428] Setting key 2b3ccd6c13d18312f6d371c2473e01e1
Sep 28 08:37:05 dist-test-slave-w6c0 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:05.911324 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:05.892889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48691 (local address 127.0.97.62:43087)
0928 08:37:05.893318 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:05.893323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:05.894275 (+ 952us) server_negotiation.cc:407] Connection header received
0928 08:37:05.896126 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:05.896131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:05.896243 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:05.896374 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:05.897911 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:05.898813 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:05.901332 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:05.901744 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:05.901840 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:05.905663 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:05.905667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:05.905671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:05.905701 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:05.908232 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:05.908945 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:05.908953 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:05.908961 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:05.909128 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:05.909561 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:05.909568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:05.909571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:05.909847 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:05.910011 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:05.910702 (+ 691us) server_negotiation.cc:299] Negotiation successful
0928 08:37:05.910935 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":213,"threads_started":1}
I20260928 08:37:05.912266 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:05.893445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43087 (local address 127.0.97.1:48691)
0928 08:37:05.894119 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:05.894154 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:05.895840 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:05.896603 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:05.896611 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:05.897149 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:05.897745 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:05.897769 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:05.898978 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:05.898984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:05.901186 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:05.901205 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:05.901966 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:05.901975 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:05.902305 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:05.903051 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:05.903089 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:05.905444 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:05.908378 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:05.908387 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:05.908402 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:05.908811 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:05.909252 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:05.909258 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:05.909261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:05.909451 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:05.909989 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:05.909996 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:05.910309 (+ 313us) client_negotiation.cc:769] Sending connection context
0928 08:37:05.910612 (+ 303us) client_negotiation.cc:240] Negotiation successful
0928 08:37:05.911141 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":261,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:37:05.913946 1152 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43087
I20260928 08:37:05.914238 1152 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:05.915040 1152 heartbeater.cc:511] Master 127.0.97.62:43087 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 08:37:05.917436 963 ts_manager.cc:194] Registered new tserver with Master: a1148a7146054140869bd9f706eb33de (127.0.97.1:39927)
I20260928 08:37:05.918965 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:37:05.938277 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43087
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:06.254981 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:06.255246 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:06.255404 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:06.262843 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:06.262948 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:06.263118 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:37:06.272982 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43087
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:06.274302 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:06.275784 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:06.288559 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:06.288604 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:06.288847 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:06.289412 1158 server_base.cc:1034] running on GCE node
I20260928 08:37:06.290117 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:06.291059 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:06.292287 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790584626292257 us; error 46 us; skew 500 ppm
I20260928 08:37:06.292598 1158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:06 dist-test-slave-w6c0 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:06.295653 1158 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:37:06.297019 1158 webserver.cc:460] Webserver started at http://127.0.97.2:46765/ using document root <none> and password file <none>
I20260928 08:37:06.297672 1158 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:06.297812 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:06.298110 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:06.300943 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "e276d440034646afaf751984b988ec64"
format_stamp: "Formatted at 2026-09-28 08:37:06 on dist-test-slave-w6c0"
server_key: "0527fbb6ee01b5377e47486bece5a2f6"
server_key_iv: "0c24bfe221f7d195785d32f80a0448d9"
server_key_version: "encryptionkey@0"
I20260928 08:37:06.301697 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e276d440034646afaf751984b988ec64"
format_stamp: "Formatted at 2026-09-28 08:37:06 on dist-test-slave-w6c0"
server_key: "0527fbb6ee01b5377e47486bece5a2f6"
server_key_iv: "0c24bfe221f7d195785d32f80a0448d9"
server_key_version: "encryptionkey@0"
I20260928 08:37:06.306641 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 08:37:06.310027 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:06.311303 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:37:06.311491 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "e276d440034646afaf751984b988ec64"
format_stamp: "Formatted at 2026-09-28 08:37:06 on dist-test-slave-w6c0"
server_key: "0527fbb6ee01b5377e47486bece5a2f6"
server_key_iv: "0c24bfe221f7d195785d32f80a0448d9"
server_key_version: "encryptionkey@0"
I20260928 08:37:06.311688 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:06.331199 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:06.335366 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:06.335676 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:06.336624 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:06.337987 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:06.338093 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:06.338240 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:06.338315 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:06.371605 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46511
I20260928 08:37:06.371688 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46511 every 8 connection(s)
I20260928 08:37:06.372952 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:37:06.374408 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 1158
I20260928 08:37:06.374632 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:37:06.375154 388 external_mini_cluster.cc:1428] Setting key 2f0dd19cc42b9f1d546d6241c6cf88dc
Sep 28 08:37:06 dist-test-slave-w6c0 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 08:37:06.395583 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:06.375760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53513 (local address 127.0.97.62:43087)
0928 08:37:06.376008 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:06.376013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:06.377337 (+ 1324us) server_negotiation.cc:407] Connection header received
0928 08:37:06.379208 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:06.379213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:06.379354 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:06.379446 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:06.381523 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.382338 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.384279 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.384692 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.384839 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:06.388700 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:06.388705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:06.388708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:06.388740 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:06.392107 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:06.392979 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:06.392984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:06.392990 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:06.393170 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:06.393728 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:06.393734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:06.393737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:06.394095 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:06.394269 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:06.395166 (+ 897us) server_negotiation.cc:299] Negotiation successful
0928 08:37:06.395346 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:37:06.397220 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:06.376328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43087 (local address 127.0.97.2:53513)
0928 08:37:06.377184 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:06.377233 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:06.378958 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:06.379736 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:06.379751 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:06.380535 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:06.381343 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:06.381371 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.382556 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.382568 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:06.384131 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:06.384143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.384918 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.384928 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:06.385153 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:06.385844 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:06.385869 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:06.388474 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:06.392249 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:06.392257 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:06.392275 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:06.392847 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:06.393289 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:06.393301 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:06.393305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:06.393606 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:06.394230 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:06.394238 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:06.394710 (+ 472us) client_negotiation.cc:769] Sending connection context
0928 08:37:06.395018 (+ 308us) client_negotiation.cc:240] Negotiation successful
0928 08:37:06.395640 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":314,"threads_started":1}
I20260928 08:37:06.398838 1286 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43087
I20260928 08:37:06.399207 1286 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:06.400087 1286 heartbeater.cc:511] Master 127.0.97.62:43087 requested a full tablet report, sending...
I20260928 08:37:06.401482 963 ts_manager.cc:194] Registered new tserver with Master: e276d440034646afaf751984b988ec64 (127.0.97.2:46511)
I20260928 08:37:06.402284 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:37:06.418287 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43087
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:06.729533 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:06.729791 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:06.729954 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:06.737210 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:06.737303 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:06.737484 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:37:06.749565 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43087
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:06.751430 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:06.752743 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:06.765484 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:06.765610 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:06.765630 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:06.766543 1291 server_base.cc:1034] running on GCE node
I20260928 08:37:06.767107 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:06.767970 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:06.769184 1291 hybrid_clock.cc:648] HybridClock initialized: now 1790584626769153 us; error 25 us; skew 500 ppm
I20260928 08:37:06.769474 1291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:06 dist-test-slave-w6c0 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:06.772519 1291 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:37:06.773897 1291 webserver.cc:460] Webserver started at http://127.0.97.3:38689/ using document root <none> and password file <none>
I20260928 08:37:06.774576 1291 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:06.774713 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:06.775012 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:06.777665 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "7414dbc098464bfa9d259af08530760d"
format_stamp: "Formatted at 2026-09-28 08:37:06 on dist-test-slave-w6c0"
server_key: "96efce22ca3f89b10aaae40c707af99c"
server_key_iv: "e6ac97a628616ce4ee3bad10349bb293"
server_key_version: "encryptionkey@0"
I20260928 08:37:06.778422 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7414dbc098464bfa9d259af08530760d"
format_stamp: "Formatted at 2026-09-28 08:37:06 on dist-test-slave-w6c0"
server_key: "96efce22ca3f89b10aaae40c707af99c"
server_key_iv: "e6ac97a628616ce4ee3bad10349bb293"
server_key_version: "encryptionkey@0"
I20260928 08:37:06.783282 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 08:37:06.786482 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:06.787739 1291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:37:06.787904 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "7414dbc098464bfa9d259af08530760d"
format_stamp: "Formatted at 2026-09-28 08:37:06 on dist-test-slave-w6c0"
server_key: "96efce22ca3f89b10aaae40c707af99c"
server_key_iv: "e6ac97a628616ce4ee3bad10349bb293"
server_key_version: "encryptionkey@0"
I20260928 08:37:06.788094 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:06.801455 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:06.805023 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:06.805325 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:06.806164 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:06.807399 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:06.807469 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:06.807548 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:06.807596 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:06.839965 1291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35719
I20260928 08:37:06.840051 1418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35719 every 8 connection(s)
I20260928 08:37:06.841200 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:37:06.846246 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 1291
I20260928 08:37:06.846463 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:37:06.846992 388 external_mini_cluster.cc:1428] Setting key bcc5e408e015a39b2080ce265a50d3b6
Sep 28 08:37:06 dist-test-slave-w6c0 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:06.860318 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:06.843871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60091 (local address 127.0.97.62:43087)
0928 08:37:06.844049 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:06.844054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:06.845124 (+ 1070us) server_negotiation.cc:407] Connection header received
0928 08:37:06.847328 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:06.847333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:06.847402 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:06.847487 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:06.849353 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.850319 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.851797 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.852178 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.852270 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:06.855511 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:06.855516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:06.855519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:06.855549 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:06.857787 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:06.858408 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:06.858412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:06.858414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:06.858577 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:06.858937 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:06.858940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:06.858942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:06.859182 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:06.859285 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:06.859886 (+ 601us) server_negotiation.cc:299] Negotiation successful
0928 08:37:06.860033 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:37:06.861232 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:06.844295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43087 (local address 127.0.97.3:60091)
0928 08:37:06.844970 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:06.845009 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:06.847044 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:06.847729 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:06.847737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:06.848436 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:06.849195 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:06.849214 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.850454 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.850457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:06.851679 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:06.851689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.852351 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.852355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:06.852534 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:06.853201 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:06.853222 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:06.855342 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:06.857903 (+ 2561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:06.857909 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:06.857920 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:06.858296 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:06.858665 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:06.858669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:06.858671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:06.858840 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:06.859279 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:06.859285 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:06.859561 (+ 276us) client_negotiation.cc:769] Sending connection context
0928 08:37:06.859760 (+ 199us) client_negotiation.cc:240] Negotiation successful
0928 08:37:06.860085 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":257,"threads_started":1}
I20260928 08:37:06.862588 1419 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43087
I20260928 08:37:06.862880 1419 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:06.863747 1419 heartbeater.cc:511] Master 127.0.97.62:43087 requested a full tablet report, sending...
I20260928 08:37:06.865475 962 ts_manager.cc:194] Registered new tserver with Master: 7414dbc098464bfa9d259af08530760d (127.0.97.3:35719)
I20260928 08:37:06.866788 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60091
I20260928 08:37:06.874173 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:37:06.892999 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:06.880192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56476 (local address 127.0.97.62:43087)
0928 08:37:06.880428 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:06.880438 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:06.880793 (+ 355us) server_negotiation.cc:407] Connection header received
0928 08:37:06.880926 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:06.880929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:06.880984 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:06.881059 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:06.882434 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.883289 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.884538 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.884939 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.885081 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:06.887357 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:06.887362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:06.887369 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:06.887403 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:06.890220 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:06.891071 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:06.891080 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:06.891083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:06.891213 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:06.891613 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:06.891616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:06.891618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:06.891971 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:06.892183 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:06.892570 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 08:37:06.892717 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 08:37:06.899016 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:37:06.905604 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:37:06.922230 1152 heartbeater.cc:503] Master 127.0.97.62:43087 was elected leader, sending a full tablet report...
I20260928 08:37:06.927592 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:06.916823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59694 (local address 127.0.97.1:39927)
0928 08:37:06.917427 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:06.917435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:06.917722 (+ 287us) server_negotiation.cc:407] Connection header received
0928 08:37:06.917835 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:06.917842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:06.918080 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:06.918992 (+ 912us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:06.919271 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.921742 (+ 2471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.924953 (+ 3211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.926471 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.926630 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:06.926858 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:06.927113 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 08:37:06.927273 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":381,"threads_started":1}
I20260928 08:37:06.927618 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:06.916580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39927 (local address 127.0.0.1:59694)
0928 08:37:06.917379 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:06.917415 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:06.917602 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:06.918331 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:06.918349 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:06.918405 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:06.918857 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:06.918875 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.921891 (+ 3016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.921896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:06.924768 (+ 2872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:06.924782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.926668 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.926674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:06.926882 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:06.926931 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 08:37:06.927056 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 08:37:06.927224 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"spinlock_wait_cycles":4992,"thread_start_us":270,"threads_started":1}
I20260928 08:37:06.931847 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 3facb3f212dc41e1b5e435d597f38b1f (DEFAULT_TABLE table=test-table [id=f76a789e6ac74b57ac52939b21a3edd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:06.932701 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:06.918208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.0.97.2:46511)
0928 08:37:06.924624 (+ 6416us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:06.924637 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:06.924669 (+ 32us) server_negotiation.cc:407] Connection header received
0928 08:37:06.924760 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:06.924768 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:06.925031 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:06.925219 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:06.926327 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.927842 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.930266 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.931657 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.931815 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:06.932012 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:06.932229 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 08:37:06.932387 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6206,"thread_start_us":6047,"threads_started":1}
I20260928 08:37:06.932740 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:06.917594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35719 (local address 127.0.0.1:47788)
0928 08:37:06.918241 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:06.918260 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:06.918405 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:06.925508 (+ 7103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:06.925513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:06.925537 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:06.925985 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:06.925997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.928202 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.928207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:06.930120 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:06.930134 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.931946 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.931953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:06.932111 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:06.932141 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 08:37:06.932205 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 08:37:06.932327 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":4,"spinlock_wait_cycles":2432,"thread_start_us":4160,"threads_started":1}
I20260928 08:37:06.933251 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:06.917712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46511 (local address 127.0.0.1:44358)
0928 08:37:06.922050 (+ 4338us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:06.922074 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:06.922185 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:06.925724 (+ 3539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:06.925731 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:06.925751 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:06.926162 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:06.926171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.928004 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.928010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:06.930124 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:06.930134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.931870 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.931880 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:06.932046 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:06.932078 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 08:37:06.932192 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 08:37:06.932322 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4178,"mutex_wait_us":26,"thread_start_us":675,"threads_started":1}
I20260928 08:37:06.933291 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:06.917998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47788 (local address 127.0.97.3:35719)
0928 08:37:06.924980 (+ 6982us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:06.924988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:06.925019 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:37:06.925109 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:06.925115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:06.925303 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:06.926189 (+ 886us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:06.926436 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.928015 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.930295 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:06.931704 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:06.932435 (+ 731us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:06.932663 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:06.932734 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 08:37:06.932879 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6801,"thread_start_us":6637,"threads_started":1}
I20260928 08:37:06.933677 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3facb3f212dc41e1b5e435d597f38b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:06.936497 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 3facb3f212dc41e1b5e435d597f38b1f (DEFAULT_TABLE table=test-table [id=f76a789e6ac74b57ac52939b21a3edd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:06.936924 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 3facb3f212dc41e1b5e435d597f38b1f (DEFAULT_TABLE table=test-table [id=f76a789e6ac74b57ac52939b21a3edd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:06.937638 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3facb3f212dc41e1b5e435d597f38b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:06.938113 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3facb3f212dc41e1b5e435d597f38b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:06.949103 1437 tablet_bootstrap.cc:492] T 3facb3f212dc41e1b5e435d597f38b1f P 7414dbc098464bfa9d259af08530760d: Bootstrap starting.
I20260928 08:37:06.951174 1438 tablet_bootstrap.cc:492] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de: Bootstrap starting.
I20260928 08:37:06.952499 1439 tablet_bootstrap.cc:492] T 3facb3f212dc41e1b5e435d597f38b1f P e276d440034646afaf751984b988ec64: Bootstrap starting.
I20260928 08:37:06.952656 1437 tablet_bootstrap.cc:654] T 3facb3f212dc41e1b5e435d597f38b1f P 7414dbc098464bfa9d259af08530760d: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:06.953917 1437 log.cc:824] T 3facb3f212dc41e1b5e435d597f38b1f P 7414dbc098464bfa9d259af08530760d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:06.955072 1438 tablet_bootstrap.cc:654] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:06.955387 1439 tablet_bootstrap.cc:654] T 3facb3f212dc41e1b5e435d597f38b1f P e276d440034646afaf751984b988ec64: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:06.956228 1438 log.cc:824] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:06.956493 1439 log.cc:824] T 3facb3f212dc41e1b5e435d597f38b1f P e276d440034646afaf751984b988ec64: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:06.957369 1437 tablet_bootstrap.cc:492] T 3facb3f212dc41e1b5e435d597f38b1f P 7414dbc098464bfa9d259af08530760d: No bootstrap required, opened a new log
I20260928 08:37:06.957628 1437 ts_tablet_manager.cc:1397] T 3facb3f212dc41e1b5e435d597f38b1f P 7414dbc098464bfa9d259af08530760d: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260928 08:37:06.958822 1438 tablet_bootstrap.cc:492] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de: No bootstrap required, opened a new log
I20260928 08:37:06.959106 1438 ts_tablet_manager.cc:1397] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 08:37:06.960397 1439 tablet_bootstrap.cc:492] T 3facb3f212dc41e1b5e435d597f38b1f P e276d440034646afaf751984b988ec64: No bootstrap required, opened a new log
I20260928 08:37:06.960621 1439 ts_tablet_manager.cc:1397] T 3facb3f212dc41e1b5e435d597f38b1f P e276d440034646afaf751984b988ec64: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 08:37:06.967006 1438 raft_consensus.cc:348] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1148a7146054140869bd9f706eb33de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39927 } } peers { permanent_uuid: "7414dbc098464bfa9d259af08530760d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35719 } } peers { permanent_uuid: "e276d440034646afaf751984b988ec64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46511 } }
I20260928 08:37:06.967427 1438 raft_consensus.cc:374] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:06.967527 1438 raft_consensus.cc:729] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1148a7146054140869bd9f706eb33de, State: Initialized, Role: FOLLOWER
I20260928 08:37:06.968138 1438 consensus_queue.cc:260] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1148a7146054140869bd9f706eb33de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39927 } } peers { permanent_uuid: "7414dbc098464bfa9d259af08530760d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35719 } } peers { permanent_uuid: "e276d440034646afaf751984b988ec64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46511 } }
I20260928 08:37:06.968276 1437 raft_consensus.cc:348] T 3facb3f212dc41e1b5e435d597f38b1f P 7414dbc098464bfa9d259af08530760d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1148a7146054140869bd9f706eb33de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39927 } } peers { permanent_uuid: "7414dbc098464bfa9d259af08530760d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35719 } } peers { permanent_uuid: "e276d440034646afaf751984b988ec64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46511 } }
I20260928 08:37:06.968662 1437 raft_consensus.cc:374] T 3facb3f212dc41e1b5e435d597f38b1f P 7414dbc098464bfa9d259af08530760d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:06.968755 1437 raft_consensus.cc:729] T 3facb3f212dc41e1b5e435d597f38b1f P 7414dbc098464bfa9d259af08530760d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7414dbc098464bfa9d259af08530760d, State: Initialized, Role: FOLLOWER
I20260928 08:37:06.969282 1437 consensus_queue.cc:260] T 3facb3f212dc41e1b5e435d597f38b1f P 7414dbc098464bfa9d259af08530760d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1148a7146054140869bd9f706eb33de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39927 } } peers { permanent_uuid: "7414dbc098464bfa9d259af08530760d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35719 } } peers { permanent_uuid: "e276d440034646afaf751984b988ec64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46511 } }
I20260928 08:37:06.970057 1438 ts_tablet_manager.cc:1428] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260928 08:37:06.970950 1419 heartbeater.cc:503] Master 127.0.97.62:43087 was elected leader, sending a full tablet report...
I20260928 08:37:06.971607 1437 ts_tablet_manager.cc:1428] T 3facb3f212dc41e1b5e435d597f38b1f P 7414dbc098464bfa9d259af08530760d: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260928 08:37:06.971398 1439 raft_consensus.cc:348] T 3facb3f212dc41e1b5e435d597f38b1f P e276d440034646afaf751984b988ec64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1148a7146054140869bd9f706eb33de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39927 } } peers { permanent_uuid: "7414dbc098464bfa9d259af08530760d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35719 } } peers { permanent_uuid: "e276d440034646afaf751984b988ec64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46511 } }
I20260928 08:37:06.971822 1439 raft_consensus.cc:374] T 3facb3f212dc41e1b5e435d597f38b1f P e276d440034646afaf751984b988ec64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:06.971931 1439 raft_consensus.cc:729] T 3facb3f212dc41e1b5e435d597f38b1f P e276d440034646afaf751984b988ec64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e276d440034646afaf751984b988ec64, State: Initialized, Role: FOLLOWER
I20260928 08:37:06.972595 1439 consensus_queue.cc:260] T 3facb3f212dc41e1b5e435d597f38b1f P e276d440034646afaf751984b988ec64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1148a7146054140869bd9f706eb33de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39927 } } peers { permanent_uuid: "7414dbc098464bfa9d259af08530760d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35719 } } peers { permanent_uuid: "e276d440034646afaf751984b988ec64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46511 } }
I20260928 08:37:06.973644 1286 heartbeater.cc:503] Master 127.0.97.62:43087 was elected leader, sending a full tablet report...
I20260928 08:37:06.974174 1439 ts_tablet_manager.cc:1428] T 3facb3f212dc41e1b5e435d597f38b1f P e276d440034646afaf751984b988ec64: Time spent starting tablet: real 0.013s user 0.005s sys 0.010s
W20260928 08:37:07.092798 1420 tablet.cc:2369] T 3facb3f212dc41e1b5e435d597f38b1f P 7414dbc098464bfa9d259af08530760d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:37:07.125149 1287 tablet.cc:2369] T 3facb3f212dc41e1b5e435d597f38b1f P e276d440034646afaf751984b988ec64: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:37:07.142725 1153 tablet.cc:2369] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:37:07.375245 1443 raft_consensus.cc:482] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:37:07.375524 1443 raft_consensus.cc:504] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1148a7146054140869bd9f706eb33de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39927 } } peers { permanent_uuid: "7414dbc098464bfa9d259af08530760d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35719 } } peers { permanent_uuid: "e276d440034646afaf751984b988ec64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46511 } }
I20260928 08:37:07.377476 1443 leader_election.cc:288] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7414dbc098464bfa9d259af08530760d (127.0.97.3:35719), e276d440034646afaf751984b988ec64 (127.0.97.2:46511)
I20260928 08:37:07.385811 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:07.377936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37949 (local address 127.0.97.2:46511)
0928 08:37:07.378116 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:07.378123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:07.378739 (+ 616us) server_negotiation.cc:407] Connection header received
0928 08:37:07.379025 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:07.379031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:07.379121 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:07.379209 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:07.379796 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.380752 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.383717 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.384838 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.384995 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:07.385099 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:07.385426 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 08:37:07.385548 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 08:37:07.385929 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:07.377707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46511 (local address 127.0.97.1:37949)
0928 08:37:07.378628 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:07.378646 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:07.378866 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:07.379231 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:07.379236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:07.379268 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:07.379614 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:07.379624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.380879 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.380886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:07.383559 (+ 2673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:07.383573 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.385091 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.385098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:07.385259 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:07.385282 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 08:37:07.385387 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 08:37:07.385521 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":1012,"threads_started":1}
I20260928 08:37:07.385973 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:07.377829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35719 (local address 127.0.97.1:58013)
0928 08:37:07.379000 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:07.379022 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:07.379138 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:07.379615 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:07.379618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:07.379636 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:07.380004 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:07.380015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.381630 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.381635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:07.383765 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:07.383774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.385155 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.385165 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:07.385301 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:07.385324 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 08:37:07.385429 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 08:37:07.385553 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"thread_start_us":447,"threads_started":1}
I20260928 08:37:07.385833 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:07.378382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58013 (local address 127.0.97.3:35719)
0928 08:37:07.378581 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:07.378586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:07.379116 (+ 530us) server_negotiation.cc:407] Connection header received
0928 08:37:07.379327 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:07.379333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:07.379407 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:07.379470 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:07.380176 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.381221 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.383907 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.384861 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.385009 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:07.385137 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:07.385455 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 08:37:07.385574 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:37:07.386963 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3facb3f212dc41e1b5e435d597f38b1f" candidate_uuid: "a1148a7146054140869bd9f706eb33de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e276d440034646afaf751984b988ec64" is_pre_election: true
I20260928 08:37:07.387081 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3facb3f212dc41e1b5e435d597f38b1f" candidate_uuid: "a1148a7146054140869bd9f706eb33de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7414dbc098464bfa9d259af08530760d" is_pre_election: true
I20260928 08:37:07.387463 1241 raft_consensus.cc:2454] T 3facb3f212dc41e1b5e435d597f38b1f P e276d440034646afaf751984b988ec64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1148a7146054140869bd9f706eb33de in term 0.
I20260928 08:37:07.387530 1374 raft_consensus.cc:2454] T 3facb3f212dc41e1b5e435d597f38b1f P 7414dbc098464bfa9d259af08530760d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1148a7146054140869bd9f706eb33de in term 0.
I20260928 08:37:07.388167 1040 leader_election.cc:302] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1148a7146054140869bd9f706eb33de, e276d440034646afaf751984b988ec64; no voters:
I20260928 08:37:07.388569 1443 raft_consensus.cc:2781] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:37:07.388707 1443 raft_consensus.cc:482] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:37:07.388813 1443 raft_consensus.cc:3037] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:07.390987 1443 raft_consensus.cc:504] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1148a7146054140869bd9f706eb33de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39927 } } peers { permanent_uuid: "7414dbc098464bfa9d259af08530760d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35719 } } peers { permanent_uuid: "e276d440034646afaf751984b988ec64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46511 } }
I20260928 08:37:07.391633 1443 leader_election.cc:288] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de [CANDIDATE]: Term 1 election: Requested vote from peers 7414dbc098464bfa9d259af08530760d (127.0.97.3:35719), e276d440034646afaf751984b988ec64 (127.0.97.2:46511)
I20260928 08:37:07.392021 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3facb3f212dc41e1b5e435d597f38b1f" candidate_uuid: "a1148a7146054140869bd9f706eb33de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7414dbc098464bfa9d259af08530760d"
I20260928 08:37:07.392068 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3facb3f212dc41e1b5e435d597f38b1f" candidate_uuid: "a1148a7146054140869bd9f706eb33de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e276d440034646afaf751984b988ec64"
I20260928 08:37:07.392220 1374 raft_consensus.cc:3037] T 3facb3f212dc41e1b5e435d597f38b1f P 7414dbc098464bfa9d259af08530760d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:07.392253 1241 raft_consensus.cc:3037] T 3facb3f212dc41e1b5e435d597f38b1f P e276d440034646afaf751984b988ec64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:07.394644 1241 raft_consensus.cc:2454] T 3facb3f212dc41e1b5e435d597f38b1f P e276d440034646afaf751984b988ec64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1148a7146054140869bd9f706eb33de in term 1.
I20260928 08:37:07.394977 1374 raft_consensus.cc:2454] T 3facb3f212dc41e1b5e435d597f38b1f P 7414dbc098464bfa9d259af08530760d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1148a7146054140869bd9f706eb33de in term 1.
I20260928 08:37:07.395011 1040 leader_election.cc:302] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1148a7146054140869bd9f706eb33de, e276d440034646afaf751984b988ec64; no voters:
I20260928 08:37:07.395267 1443 raft_consensus.cc:2781] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:07.395736 1443 raft_consensus.cc:686] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de [term 1 LEADER]: Becoming Leader. State: Replica: a1148a7146054140869bd9f706eb33de, State: Running, Role: LEADER
I20260928 08:37:07.396344 1443 consensus_queue.cc:237] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1148a7146054140869bd9f706eb33de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39927 } } peers { permanent_uuid: "7414dbc098464bfa9d259af08530760d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35719 } } peers { permanent_uuid: "e276d440034646afaf751984b988ec64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46511 } }
I20260928 08:37:07.401033 962 catalog_manager.cc:5495] T 3facb3f212dc41e1b5e435d597f38b1f P a1148a7146054140869bd9f706eb33de reported cstate change: term changed from 0 to 1, leader changed from <none> to a1148a7146054140869bd9f706eb33de (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a1148a7146054140869bd9f706eb33de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1148a7146054140869bd9f706eb33de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7414dbc098464bfa9d259af08530760d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e276d440034646afaf751984b988ec64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46511 } health_report { overall_health: UNKNOWN } } }
Sep 28 08:37:07 dist-test-slave-w6c0 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:37:07.495005 1452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:07.480014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59698 (local address 127.0.97.1:39927)
0928 08:37:07.480486 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:07.480492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:07.480638 (+ 146us) server_negotiation.cc:407] Connection header received
0928 08:37:07.480851 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:07.480855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:07.480929 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:07.481044 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:07.482377 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.483146 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.484432 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.484911 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.485042 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:07.489041 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:07.489052 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:07.489068 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:07.489098 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:07.492255 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:07.492966 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:07.492982 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:07.492984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:07.493109 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:07.493505 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:07.493512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:07.493515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:07.493851 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:07.494152 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:07.494417 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 08:37:07.494646 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":237,"threads_started":1}
Sep 28 08:37:07 dist-test-slave-w6c0 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584627, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:37:07 dist-test-slave-w6c0 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584627, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:37:07.538918 1452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:07.525174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.0.97.1:39927)
0928 08:37:07.525437 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:07.525445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:07.525756 (+ 311us) server_negotiation.cc:407] Connection header received
0928 08:37:07.525979 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:07.525988 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:07.526056 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:07.526168 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:07.527547 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.528243 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.529489 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.529912 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.530046 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:07.533773 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:07.533778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:07.533782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:07.533812 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:07.536338 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:07.536999 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:07.537006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:07.537010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:07.537144 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:07.537506 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:07.537511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:07.537514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:07.537776 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:07.538024 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:07.538324 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 08:37:07.538591 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Sep 28 08:37:07 dist-test-slave-w6c0 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:37:07 dist-test-slave-w6c0 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:37:07.582122 1452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:07.568547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.0.97.1:39927)
0928 08:37:07.568801 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:07.568809 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:07.569070 (+ 261us) server_negotiation.cc:407] Connection header received
0928 08:37:07.569283 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:07.569287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:07.569363 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:07.569471 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:07.571307 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.572000 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.573211 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.573654 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.573788 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:07.577695 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:07.577700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:07.577703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:07.577733 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:07.580106 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:07.580640 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:07.580643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:07.580646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:07.580769 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:07.581068 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:07.581071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:07.581073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:07.581318 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:07.581489 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:07.581715 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 08:37:07.581867 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260928 08:37:07.584069 1067 tablet_service.cc:2908] Rejecting scan request for tablet 3facb3f212dc41e1b5e435d597f38b1f: Uninitialized: clean time has not yet been initialized
I20260928 08:37:07.586952 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 1024
I20260928 08:37:07.598770 1452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:07.587922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59732 (local address 127.0.97.1:39927)
0928 08:37:07.588092 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:07.588096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:07.588433 (+ 337us) server_negotiation.cc:407] Connection header received
0928 08:37:07.588548 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:07.588552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:07.588621 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:07.588704 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:07.589704 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.590394 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.591738 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.592209 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.592300 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:07.594259 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:07.594264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:07.594267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:07.594293 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:07.596749 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:07.597320 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:07.597324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:07.597326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:07.597427 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:07.597724 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:07.597727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:07.597729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:07.597951 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:07.598115 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:07.598339 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 08:37:07.598481 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 08:37:07.599372 1147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:07.689658 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 1024
I20260928 08:37:07.706785 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 1158
Sep 28 08:37:07 dist-test-slave-w6c0 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:37:07.720844 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:07.707590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44364 (local address 127.0.97.2:46511)
0928 08:37:07.707774 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:07.707778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:07.707796 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:37:07.707876 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:07.707880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:07.707943 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:07.708037 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:07.709148 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.709852 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.711207 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.711594 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.711721 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:07.715662 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:07.715676 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:07.715692 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:07.715724 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:07.718488 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:07.719185 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:07.719193 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:07.719195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:07.719313 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:07.719607 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:07.719610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:07.719612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:07.719981 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:07.720189 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:07.720419 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 08:37:07.720573 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:37:07.721436 1281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:07.807834 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 1158
I20260928 08:37:07.825217 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 1291
Sep 28 08:37:07 dist-test-slave-w6c0 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:37:07.839214 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:07.825971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.0.97.3:35719)
0928 08:37:07.826133 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:07.826137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:07.826155 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:37:07.826215 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:07.826218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:07.826281 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:07.826368 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:07.827625 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.828350 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.829525 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:07.829903 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:07.829992 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:07.834148 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:07.834161 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:07.834179 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:07.834211 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:07.836843 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:07.837420 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:07.837427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:07.837429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:07.837537 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:07.837862 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:07.837865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:07.837867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:07.838195 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:07.838412 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:07.838671 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 08:37:07.838839 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 08:37:07.839984 1414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:07.928961 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 1291
I20260928 08:37:07.946646 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 931
I20260928 08:37:07.947822 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:08.057694 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 931
2026-09-28T08:37:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4419 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:37:08 dist-test-slave-w6c0 krb5kdc[1468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:37:08 dist-test-slave-w6c0 krb5kdc[1468](info): set up 2 sockets
Sep 28 08:37:08 dist-test-slave-w6c0 krb5kdc[1468](info): commencing operation
krb5kdc: starting...
W20260928 08:37:09.150637 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:37:09 dist-test-slave-w6c0 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:37:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:37:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:37:09.330351 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33851
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:09.645256 1481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:09.645597 1481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:09.645690 1481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:09.653029 1481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:37:09.653144 1481 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 08:37:09.653185 1481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:09.653240 1481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:09.653290 1481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:37:09.653345 1481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:37:09.663218 1481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33851
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:09.664446 1481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:09.666127 1481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:09.678524 1489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:09.678583 1486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:09.678606 1487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:09.679329 1481 server_base.cc:1034] running on GCE node
I20260928 08:37:09.680071 1481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:09.681584 1481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:09.682850 1481 hybrid_clock.cc:648] HybridClock initialized: now 1790584629682792 us; error 56 us; skew 500 ppm
I20260928 08:37:09.683203 1481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:09 dist-test-slave-w6c0 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:09.686378 1481 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:37:09.687922 1481 webserver.cc:460] Webserver started at http://127.0.97.62:40245/ using document root <none> and password file <none>
I20260928 08:37:09.688581 1481 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:09.688664 1481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:09.688930 1481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:09.691852 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "adcbde99d7854b5eb1aa9d6bb8dd1279"
format_stamp: "Formatted at 2026-09-28 08:37:09 on dist-test-slave-w6c0"
server_key: "172e5436308bec7439274a23a500967e"
server_key_iv: "c726892ec2853ced38b172d1c2ebccf0"
server_key_version: "encryptionkey@0"
I20260928 08:37:09.692610 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "adcbde99d7854b5eb1aa9d6bb8dd1279"
format_stamp: "Formatted at 2026-09-28 08:37:09 on dist-test-slave-w6c0"
server_key: "172e5436308bec7439274a23a500967e"
server_key_iv: "c726892ec2853ced38b172d1c2ebccf0"
server_key_version: "encryptionkey@0"
I20260928 08:37:09.697602 1481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260928 08:37:09.701129 1496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:09.702481 1481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 08:37:09.702713 1481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "adcbde99d7854b5eb1aa9d6bb8dd1279"
format_stamp: "Formatted at 2026-09-28 08:37:09 on dist-test-slave-w6c0"
server_key: "172e5436308bec7439274a23a500967e"
server_key_iv: "c726892ec2853ced38b172d1c2ebccf0"
server_key_version: "encryptionkey@0"
I20260928 08:37:09.702885 1481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:09.717072 1481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:09.721138 1481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:09.721359 1481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:09.743010 1481 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33851
I20260928 08:37:09.743028 1557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33851 every 8 connection(s)
I20260928 08:37:09.744400 1481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:37:09.747531 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 1481
I20260928 08:37:09.747747 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:37:09.748522 1558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:09.748597 388 external_mini_cluster.cc:1428] Setting key 3d047e1c1aa1c65e130d60098f2abc54
I20260928 08:37:09.762883 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279: Bootstrap starting.
Sep 28 08:37:09 dist-test-slave-w6c0 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:09.766750 1558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:09.768076 1558 log.cc:824] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:09.771288 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279: No bootstrap required, opened a new log
I20260928 08:37:09.772315 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:09.750674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.0.97.62:33851)
0928 08:37:09.751464 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:09.751475 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:09.751521 (+ 46us) server_negotiation.cc:407] Connection header received
0928 08:37:09.753241 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:09.753262 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:09.753638 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:09.753966 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:09.755749 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:09.757004 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:09.758759 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:09.759429 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:09.759563 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:09.764193 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:09.764205 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:09.764224 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:09.764289 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:09.768135 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:09.768963 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:09.768977 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:09.768986 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:09.769206 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:09.769701 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:09.769705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:09.769707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:09.770200 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:09.770417 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:09.770913 (+ 496us) server_negotiation.cc:299] Negotiation successful
0928 08:37:09.771298 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":363,"threads_started":1}
I20260928 08:37:09.779263 1558 raft_consensus.cc:348] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcbde99d7854b5eb1aa9d6bb8dd1279" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } }
I20260928 08:37:09.779618 1558 raft_consensus.cc:374] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:09.779721 1558 raft_consensus.cc:729] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adcbde99d7854b5eb1aa9d6bb8dd1279, State: Initialized, Role: FOLLOWER
I20260928 08:37:09.780242 1558 consensus_queue.cc:260] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcbde99d7854b5eb1aa9d6bb8dd1279" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } }
I20260928 08:37:09.780479 1558 raft_consensus.cc:388] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:37:09.780614 1558 raft_consensus.cc:482] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:37:09.780774 1558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:09.783037 1558 raft_consensus.cc:504] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcbde99d7854b5eb1aa9d6bb8dd1279" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } }
I20260928 08:37:09.783517 1558 leader_election.cc:302] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adcbde99d7854b5eb1aa9d6bb8dd1279; no voters:
I20260928 08:37:09.784070 1558 leader_election.cc:288] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:37:09.784196 1563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:09.784915 1563 raft_consensus.cc:686] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279 [term 1 LEADER]: Becoming Leader. State: Replica: adcbde99d7854b5eb1aa9d6bb8dd1279, State: Running, Role: LEADER
I20260928 08:37:09.785410 1563 consensus_queue.cc:237] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcbde99d7854b5eb1aa9d6bb8dd1279" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } }
I20260928 08:37:09.786087 1558 sys_catalog.cc:564] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:37:09.788311 1565 sys_catalog.cc:455] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adcbde99d7854b5eb1aa9d6bb8dd1279. Latest consensus state: current_term: 1 leader_uuid: "adcbde99d7854b5eb1aa9d6bb8dd1279" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcbde99d7854b5eb1aa9d6bb8dd1279" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } } }
I20260928 08:37:09.788759 1565 sys_catalog.cc:458] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:09.790802 1572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:37:09.790689 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adcbde99d7854b5eb1aa9d6bb8dd1279" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcbde99d7854b5eb1aa9d6bb8dd1279" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } } }
I20260928 08:37:09.791054 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:09.795931 1572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:37:09.804644 1572 catalog_manager.cc:1347] Generated new cluster ID: 4a65885b713d4a369dfbec426060bfdb
I20260928 08:37:09.804774 1572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:37:09.817241 1572 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:37:09.818557 1572 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:37:09.827296 1572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279: Generated new TSK 0
I20260928 08:37:09.828153 1572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:37:09.885985 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33851
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:10.237366 1584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:10.237689 1584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:10.237857 1584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:10.245139 1584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:10.245288 1584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:10.245435 1584 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:37:10.245579 1584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:37:10.255613 1584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:10.256907 1584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:10.258242 1584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:10.271067 1589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:10.271101 1590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:10.272076 1584 server_base.cc:1034] running on GCE node
W20260928 08:37:10.271229 1592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:10.272894 1584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:10.273962 1584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:10.275192 1584 hybrid_clock.cc:648] HybridClock initialized: now 1790584630275126 us; error 43 us; skew 500 ppm
I20260928 08:37:10.275522 1584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:10 dist-test-slave-w6c0 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:10.284607 1584 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:37:10.286065 1584 webserver.cc:460] Webserver started at http://127.0.97.1:36047/ using document root <none> and password file <none>
I20260928 08:37:10.286846 1584 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:10.286971 1584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:10.287289 1584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:10.290268 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc"
format_stamp: "Formatted at 2026-09-28 08:37:10 on dist-test-slave-w6c0"
server_key: "72c9c9b03ceab336a33cf13e4c7fcab1"
server_key_iv: "ceb96d7ee6bcb6c88cc0c75c59cbd5e4"
server_key_version: "encryptionkey@0"
I20260928 08:37:10.291183 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc"
format_stamp: "Formatted at 2026-09-28 08:37:10 on dist-test-slave-w6c0"
server_key: "72c9c9b03ceab336a33cf13e4c7fcab1"
server_key_iv: "ceb96d7ee6bcb6c88cc0c75c59cbd5e4"
server_key_version: "encryptionkey@0"
I20260928 08:37:10.296262 1584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 08:37:10.299690 1599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:10.301023 1584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:37:10.301199 1584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc"
format_stamp: "Formatted at 2026-09-28 08:37:10 on dist-test-slave-w6c0"
server_key: "72c9c9b03ceab336a33cf13e4c7fcab1"
server_key_iv: "ceb96d7ee6bcb6c88cc0c75c59cbd5e4"
server_key_version: "encryptionkey@0"
I20260928 08:37:10.301416 1584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:10.319265 1584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:10.323288 1584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:10.323577 1584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:10.326263 1584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:10.326366 1584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:10.326583 1584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:10.326673 1584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:37:10 dist-test-slave-w6c0 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:10.349615 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:10.326847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33079 (local address 127.0.97.62:33851)
0928 08:37:10.329453 (+ 2606us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:10.329460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:10.329485 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:37:10.330842 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:10.330847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:10.330946 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:10.331077 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:10.332623 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:10.333692 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:10.336325 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:10.336738 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:10.336876 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:10.341890 (+ 5014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:10.341899 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:10.341906 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:10.341955 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:10.345966 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:10.346976 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:10.346982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:10.346985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:10.347108 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:10.347736 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:10.347742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:10.347747 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:10.348156 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:10.348321 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:10.349038 (+ 717us) server_negotiation.cc:299] Negotiation successful
0928 08:37:10.349258 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2461,"thread_start_us":2360,"threads_started":1}
I20260928 08:37:10.350689 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:10.327471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33851 (local address 127.0.97.1:33079)
0928 08:37:10.328345 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:10.328397 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:10.330476 (+ 2079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:10.331321 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:10.331329 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:10.331840 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:10.332431 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:10.332456 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:10.333881 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:10.333885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:10.336170 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:10.336183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:10.336968 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:10.336978 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:10.337205 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:10.338012 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:10.338036 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:10.341607 (+ 3571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:10.346189 (+ 4582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:10.346200 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:10.346226 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:10.346836 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:10.347278 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:10.347285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:10.347290 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:10.347609 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:10.348295 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:10.348304 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:10.348666 (+ 362us) client_negotiation.cc:769] Sending connection context
0928 08:37:10.348923 (+ 257us) client_negotiation.cc:240] Negotiation successful
0928 08:37:10.349425 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":394,"threads_started":1}
I20260928 08:37:10.373723 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:10.356982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33851 (local address 127.0.97.1:34147)
0928 08:37:10.357596 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:10.357615 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:10.357776 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:10.358113 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:10.358117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:10.358686 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:10.359305 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:10.359326 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:10.360446 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:10.360453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:10.362838 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:10.362861 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:10.364333 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:10.364345 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:10.364604 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:10.365483 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:10.365540 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:10.366961 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:10.371268 (+ 4307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:10.371272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:10.371274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:10.371695 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:10.372138 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:10.372142 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:10.372144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:10.372256 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:10.372916 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:10.372920 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:10.373067 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 08:37:10.373173 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 08:37:10.373399 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":367,"threads_started":1}
I20260928 08:37:10.374354 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:10.356973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34147 (local address 127.0.97.62:33851)
0928 08:37:10.357137 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:10.357141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:10.357711 (+ 570us) server_negotiation.cc:407] Connection header received
0928 08:37:10.357905 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:10.357911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:10.357987 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:10.358091 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:10.359531 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:10.360287 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:10.363059 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:10.363856 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:10.363967 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:10.367195 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:10.367205 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:10.367213 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:10.367264 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:10.371119 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:10.371836 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:10.371844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:10.371850 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:10.372015 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:10.372375 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:10.372383 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:10.372389 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:10.372790 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:10.372973 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:10.373345 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 08:37:10.373646 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:37:10.384051 1584 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35691
I20260928 08:37:10.384094 1719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35691 every 8 connection(s)
I20260928 08:37:10.385277 1584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:37:10.386991 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 1584
I20260928 08:37:10.387213 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:37:10.387751 1720 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33851
I20260928 08:37:10.387996 1720 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:10.388073 388 external_mini_cluster.cc:1428] Setting key 58e3e39a16c0991c8916db146655e09b
I20260928 08:37:10.388646 1720 heartbeater.cc:511] Master 127.0.97.62:33851 requested a full tablet report, sending...
I20260928 08:37:10.390944 1513 ts_manager.cc:194] Registered new tserver with Master: 80fcad2ad6cf46b7b3798fdbcc8fd2cc (127.0.97.1:35691)
I20260928 08:37:10.393153 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33079
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:37:10.428092 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33851
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:10.751294 1724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:10.751653 1724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:10.751796 1724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:10.763346 1724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:10.763501 1724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:10.763640 1724 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:37:10.763758 1724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:37:10.774638 1724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:10.775893 1724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:10.777149 1724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:10.790263 1729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:10.790297 1730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:10.790337 1732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:10.791571 1724 server_base.cc:1034] running on GCE node
I20260928 08:37:10.792233 1724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:10.793203 1724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:10.794457 1724 hybrid_clock.cc:648] HybridClock initialized: now 1790584630794378 us; error 60 us; skew 500 ppm
I20260928 08:37:10.794795 1724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:10 dist-test-slave-w6c0 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:10.797888 1724 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:37:10.799496 1724 webserver.cc:460] Webserver started at http://127.0.97.2:37701/ using document root <none> and password file <none>
I20260928 08:37:10.800172 1724 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:10.800256 1724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:10.800599 1724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:10.803411 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "7a0af96e414c403faeaf394b27744a76"
format_stamp: "Formatted at 2026-09-28 08:37:10 on dist-test-slave-w6c0"
server_key: "4708fbaf874da71f31fd8b54f6cff581"
server_key_iv: "9096c1c164c81e27ba52f49752643f00"
server_key_version: "encryptionkey@0"
I20260928 08:37:10.804195 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7a0af96e414c403faeaf394b27744a76"
format_stamp: "Formatted at 2026-09-28 08:37:10 on dist-test-slave-w6c0"
server_key: "4708fbaf874da71f31fd8b54f6cff581"
server_key_iv: "9096c1c164c81e27ba52f49752643f00"
server_key_version: "encryptionkey@0"
I20260928 08:37:10.809288 1724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:37:10.812517 1739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:10.813824 1724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:37:10.813993 1724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "7a0af96e414c403faeaf394b27744a76"
format_stamp: "Formatted at 2026-09-28 08:37:10 on dist-test-slave-w6c0"
server_key: "4708fbaf874da71f31fd8b54f6cff581"
server_key_iv: "9096c1c164c81e27ba52f49752643f00"
server_key_version: "encryptionkey@0"
I20260928 08:37:10.814206 1724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:10.832741 1724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:10.836869 1724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:10.837145 1724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:10.839823 1724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:10.839933 1724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:10.840097 1724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:10.840171 1724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:37:10 dist-test-slave-w6c0 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:10.862632 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:10.840272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50009 (local address 127.0.97.62:33851)
0928 08:37:10.840504 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:10.840514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:10.841903 (+ 1389us) server_negotiation.cc:407] Connection header received
0928 08:37:10.843994 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:10.844000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:10.844086 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:10.844215 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:10.847036 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:10.848023 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:10.850386 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:10.850980 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:10.851127 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:10.855673 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:10.855678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:10.855681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:10.855714 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:10.858798 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:10.860040 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:10.860046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:10.860049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:10.860200 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:10.860771 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:10.860779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:10.860785 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:10.861199 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:10.861354 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:10.862033 (+ 679us) server_negotiation.cc:299] Negotiation successful
0928 08:37:10.862244 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:37:10.864343 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:10.840848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33851 (local address 127.0.97.2:50009)
0928 08:37:10.841718 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:10.841765 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:10.843657 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:10.844712 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:10.844725 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:10.845665 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:10.846788 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:10.846825 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:10.848246 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:10.848250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:10.850196 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:10.850229 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:10.851483 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:10.851492 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:10.851789 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:10.852725 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:10.852746 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:10.855421 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:10.858972 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:10.858982 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:10.859000 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:10.859846 (+ 846us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:10.860360 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:10.860368 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:10.860377 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:10.860634 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:10.861328 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:10.861336 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:10.861690 (+ 354us) client_negotiation.cc:769] Sending connection context
0928 08:37:10.861942 (+ 252us) client_negotiation.cc:240] Negotiation successful
0928 08:37:10.862460 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":387,"threads_started":1}
I20260928 08:37:10.887480 1724 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43583
I20260928 08:37:10.887509 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:10.870342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47835 (local address 127.0.97.62:33851)
0928 08:37:10.870589 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:10.870597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:10.871065 (+ 468us) server_negotiation.cc:407] Connection header received
0928 08:37:10.871315 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:10.871322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:10.871416 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:10.871505 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:10.873272 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:10.874276 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:10.877317 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:10.878040 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:10.878189 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:10.880270 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:10.880276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:10.880281 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:10.880321 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:10.884228 (+ 3907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:10.885075 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:10.885083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:10.885089 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:10.885250 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:10.885640 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:10.885648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:10.885654 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:10.886113 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:10.886319 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:10.886814 (+ 495us) server_negotiation.cc:299] Negotiation successful
0928 08:37:10.887055 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 08:37:10.887522 1858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43583 every 8 connection(s)
I20260928 08:37:10.887646 1813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:10.870227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33851 (local address 127.0.97.2:47835)
0928 08:37:10.870947 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:10.870978 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:10.871145 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:10.871676 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:10.871688 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:10.872494 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:10.873040 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:10.873058 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:10.874408 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:10.874415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:10.877090 (+ 2675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:10.877117 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:10.878221 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:10.878225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:10.878347 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:10.879115 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:10.879145 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:10.880064 (+ 919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:10.884372 (+ 4308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:10.884377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:10.884382 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:10.884903 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:10.885363 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:10.885366 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:10.885369 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:10.885502 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:10.886241 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:10.886246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:10.886444 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 08:37:10.886641 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 08:37:10.887033 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":383,"threads_started":1}
I20260928 08:37:10.889232 1724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:37:10.891286 1859 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33851
I20260928 08:37:10.891508 1859 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:10.892210 1859 heartbeater.cc:511] Master 127.0.97.62:33851 requested a full tablet report, sending...
I20260928 08:37:10.893441 1513 ts_manager.cc:194] Registered new tserver with Master: 7a0af96e414c403faeaf394b27744a76 (127.0.97.2:43583)
I20260928 08:37:10.894169 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50009
I20260928 08:37:10.895543 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 1724
I20260928 08:37:10.895732 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:37:10.896215 388 external_mini_cluster.cc:1428] Setting key 6d22d185ad678d351bd7a17edce5dfab
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:37:10.935920 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33851
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:11.258011 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:11.258291 1863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:11.258478 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:11.265650 1863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:11.265758 1863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:11.265971 1863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:37:11.266117 1863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:37:11.276094 1863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:11.277352 1863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:11.278712 1863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:11.291594 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:11.291947 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:11.292004 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:11.292426 1863 server_base.cc:1034] running on GCE node
I20260928 08:37:11.293020 1863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:11.294036 1863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:11.295275 1863 hybrid_clock.cc:648] HybridClock initialized: now 1790584631295221 us; error 51 us; skew 500 ppm
I20260928 08:37:11.295567 1863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:11 dist-test-slave-w6c0 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:11.298677 1863 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:37:11.300062 1863 webserver.cc:460] Webserver started at http://127.0.97.3:41925/ using document root <none> and password file <none>
I20260928 08:37:11.300733 1863 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:11.300843 1863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:11.301232 1863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:11.304165 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "143aa94c7705441c9dbd0f56d21f56b9"
format_stamp: "Formatted at 2026-09-28 08:37:11 on dist-test-slave-w6c0"
server_key: "fdb4275e527bc3e0e4ec80c6635d7a6f"
server_key_iv: "20a48a5c04e7f70db629352956bb53e3"
server_key_version: "encryptionkey@0"
I20260928 08:37:11.304919 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "143aa94c7705441c9dbd0f56d21f56b9"
format_stamp: "Formatted at 2026-09-28 08:37:11 on dist-test-slave-w6c0"
server_key: "fdb4275e527bc3e0e4ec80c6635d7a6f"
server_key_iv: "20a48a5c04e7f70db629352956bb53e3"
server_key_version: "encryptionkey@0"
I20260928 08:37:11.309937 1863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:37:11.313329 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:11.314690 1863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:37:11.314857 1863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "143aa94c7705441c9dbd0f56d21f56b9"
format_stamp: "Formatted at 2026-09-28 08:37:11 on dist-test-slave-w6c0"
server_key: "fdb4275e527bc3e0e4ec80c6635d7a6f"
server_key_iv: "20a48a5c04e7f70db629352956bb53e3"
server_key_version: "encryptionkey@0"
I20260928 08:37:11.315059 1863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:11.341100 1863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:11.345610 1863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:11.345912 1863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:11.348642 1863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:11.348757 1863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:11.348920 1863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:11.349007 1863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:37:11 dist-test-slave-w6c0 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:11.372651 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:11.349288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35507 (local address 127.0.97.62:33851)
0928 08:37:11.349476 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:11.349483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:11.350880 (+ 1397us) server_negotiation.cc:407] Connection header received
0928 08:37:11.352888 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:11.352898 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:11.353010 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:11.353134 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:11.355393 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.356457 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.359010 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.359729 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.359912 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:11.364486 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:11.364496 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:11.364504 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:11.364561 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:11.369004 (+ 4443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:11.369749 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:11.369758 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:11.369764 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:11.369933 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:11.370528 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:11.370536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:11.370540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:11.370949 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:11.371112 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:11.372043 (+ 931us) server_negotiation.cc:299] Negotiation successful
0928 08:37:11.372262 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:37:11.373637 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:11.349833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33851 (local address 127.0.97.3:35507)
0928 08:37:11.350710 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:11.350757 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:11.352588 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:11.353416 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:11.353431 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:11.354187 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:11.355167 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:11.355195 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.356607 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.356612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:11.358834 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:11.358854 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.360017 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.360026 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:11.360346 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:11.361203 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:11.361224 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:11.364217 (+ 2993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:11.369160 (+ 4943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:11.369167 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:11.369180 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:11.369620 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:11.370071 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:11.370078 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:11.370084 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:11.370376 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:11.371088 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:11.371096 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:11.371627 (+ 531us) client_negotiation.cc:769] Sending connection context
0928 08:37:11.371926 (+ 299us) client_negotiation.cc:240] Negotiation successful
0928 08:37:11.372459 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":374,"threads_started":1}
I20260928 08:37:11.396066 1720 heartbeater.cc:503] Master 127.0.97.62:33851 was elected leader, sending a full tablet report...
I20260928 08:37:11.397203 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:11.380176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43031 (local address 127.0.97.62:33851)
0928 08:37:11.380414 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:11.380423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:11.381073 (+ 650us) server_negotiation.cc:407] Connection header received
0928 08:37:11.381231 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:11.381240 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:11.381368 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:11.381506 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:11.382390 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.383858 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.385860 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.386867 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.387028 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:11.389182 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:11.389192 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:11.389200 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:11.389251 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:11.393538 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:11.394560 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:11.394569 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:11.394575 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:11.394763 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:11.395282 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:11.395302 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:11.395308 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:11.395700 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:11.395864 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:11.396573 (+ 709us) server_negotiation.cc:299] Negotiation successful
0928 08:37:11.396821 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 08:37:11.397533 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:11.380319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33851 (local address 127.0.97.3:43031)
0928 08:37:11.380956 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:11.380975 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:11.381105 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:11.381527 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:11.381530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:11.381960 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:11.382235 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:11.382243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.384026 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.384031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:11.385669 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:11.385683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.387085 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.387094 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:11.387306 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:11.388010 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:11.388032 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:11.388962 (+ 930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:11.393754 (+ 4792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:11.393762 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:11.393768 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:11.394399 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:11.394915 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:11.394922 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:11.394929 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:11.395123 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:11.395945 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:11.395952 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:11.396217 (+ 265us) client_negotiation.cc:769] Sending connection context
0928 08:37:11.396433 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 08:37:11.396889 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":367,"threads_started":1}
I20260928 08:37:11.412165 1863 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36413
I20260928 08:37:11.412235 1997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36413 every 8 connection(s)
I20260928 08:37:11.413371 1863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:37:11.415323 1998 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33851
I20260928 08:37:11.415506 1998 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:11.416064 1998 heartbeater.cc:511] Master 127.0.97.62:33851 requested a full tablet report, sending...
I20260928 08:37:11.417266 1513 ts_manager.cc:194] Registered new tserver with Master: 143aa94c7705441c9dbd0f56d21f56b9 (127.0.97.3:36413)
I20260928 08:37:11.418118 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35507
I20260928 08:37:11.423288 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 1863
I20260928 08:37:11.423429 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:37:11.423786 388 external_mini_cluster.cc:1428] Setting key d79e0d747851e9cacec6aaec49775045
I20260928 08:37:11.426764 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:37:11 dist-test-slave-w6c0 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584631, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:37:11 dist-test-slave-w6c0 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584631, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:11.470738 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:11.456779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.0.97.62:33851)
0928 08:37:11.457090 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:11.457096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:11.457265 (+ 169us) server_negotiation.cc:407] Connection header received
0928 08:37:11.457500 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:11.457506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:11.457582 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:11.457692 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:11.459116 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.459837 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.461079 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.461524 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.461658 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:11.465455 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:11.465461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:11.465464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:11.465498 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:11.468191 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:11.468810 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:11.468822 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:11.468826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:11.468951 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:11.469331 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:11.469337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:11.469340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:11.469612 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:11.469854 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:11.470140 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 08:37:11.470356 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260928 08:37:11.474828 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:37:11.481717 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:37:11.505208 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:11.494896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37456 (local address 127.0.97.1:35691)
0928 08:37:11.497438 (+ 2542us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:11.497446 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:11.497471 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:37:11.497557 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:11.497563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:11.497719 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:11.497871 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:11.498655 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.500234 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.502232 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.503538 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.503670 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:11.503883 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:11.504722 (+ 839us) server_negotiation.cc:299] Negotiation successful
0928 08:37:11.504892 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2404,"thread_start_us":2263,"threads_started":1}
I20260928 08:37:11.505450 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:11.493828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36413 (local address 127.0.0.1:42310)
0928 08:37:11.494914 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:11.494956 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:11.495106 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:11.498151 (+ 3045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:11.498160 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:11.498189 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:11.498783 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:11.498794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.500701 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.500706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:11.502939 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:11.502952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.504653 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.504661 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:11.504829 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:11.504861 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 08:37:11.504948 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 08:37:11.505072 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":1,"thread_start_us":751,"threads_started":1}
I20260928 08:37:11.505757 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:11.494105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42310 (local address 127.0.97.3:36413)
0928 08:37:11.496749 (+ 2644us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:11.496756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:11.496780 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:37:11.496851 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:11.496856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:11.497073 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:11.498730 (+ 1657us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:11.499046 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.500564 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.503116 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.504393 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.504554 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:11.504780 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:11.505013 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 08:37:11.505177 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2516,"thread_start_us":2364,"threads_started":1}
I20260928 08:37:11.505936 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:11.494120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35691 (local address 127.0.0.1:37456)
0928 08:37:11.495836 (+ 1716us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:11.495851 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:11.495968 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:11.497919 (+ 1951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:11.497928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:11.497958 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:11.498418 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:11.498436 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.500380 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.500385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:11.502081 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:11.502092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.504007 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.504013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:11.504207 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:11.504276 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 08:37:11.504421 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 08:37:11.505527 (+ 1106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1590,"mutex_wait_us":3,"thread_start_us":2060,"threads_started":1}
I20260928 08:37:11.506215 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:11.495313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43583 (local address 127.0.0.1:39398)
0928 08:37:11.496204 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:11.496215 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:11.496287 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:11.497983 (+ 1696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:11.497990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:11.498019 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:11.498646 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:11.498663 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.500204 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.500209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:11.502675 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:11.502688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.504447 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.504458 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:11.504701 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:11.504735 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 08:37:11.504849 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 08:37:11.504973 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":28,"spinlock_wait_cycles":29056,"thread_start_us":272,"threads_started":1}
I20260928 08:37:11.506592 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:11.495672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.0.97.2:43583)
0928 08:37:11.497538 (+ 1866us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:11.497544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:11.497564 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:37:11.497638 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:11.497642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:11.497777 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:11.497992 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:11.498869 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.500052 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.502839 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.504151 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.504276 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:11.504463 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:11.505580 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0928 08:37:11.505889 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1739,"thread_start_us":1650,"threads_started":1}
I20260928 08:37:11.509320 1650 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5718edda114106ba2fa2e433b73e44 (DEFAULT_TABLE table=test-table [id=b15d480394d448c3b7fa0baf40a54ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:11.509488 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5718edda114106ba2fa2e433b73e44 (DEFAULT_TABLE table=test-table [id=b15d480394d448c3b7fa0baf40a54ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:11.509730 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5718edda114106ba2fa2e433b73e44 (DEFAULT_TABLE table=test-table [id=b15d480394d448c3b7fa0baf40a54ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:11.510619 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f5718edda114106ba2fa2e433b73e44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:11.510836 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f5718edda114106ba2fa2e433b73e44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:11.510838 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f5718edda114106ba2fa2e433b73e44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:11.522015 2017 tablet_bootstrap.cc:492] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Bootstrap starting.
I20260928 08:37:11.524718 2018 tablet_bootstrap.cc:492] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76: Bootstrap starting.
I20260928 08:37:11.526065 2019 tablet_bootstrap.cc:492] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9: Bootstrap starting.
I20260928 08:37:11.527078 2017 tablet_bootstrap.cc:654] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:11.528003 2018 tablet_bootstrap.cc:654] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:11.528424 2017 log.cc:824] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:11.529220 2018 log.cc:824] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:11.529263 2019 tablet_bootstrap.cc:654] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:11.530200 2019 log.cc:824] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:11.531525 2017 tablet_bootstrap.cc:492] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: No bootstrap required, opened a new log
I20260928 08:37:11.531788 2017 ts_tablet_manager.cc:1397] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 08:37:11.533105 2018 tablet_bootstrap.cc:492] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76: No bootstrap required, opened a new log
I20260928 08:37:11.533373 2018 ts_tablet_manager.cc:1397] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 08:37:11.538594 2019 tablet_bootstrap.cc:492] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9: No bootstrap required, opened a new log
I20260928 08:37:11.538838 2019 ts_tablet_manager.cc:1397] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.011s
I20260928 08:37:11.543743 2017 raft_consensus.cc:348] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } } peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } }
I20260928 08:37:11.544186 2017 raft_consensus.cc:374] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:11.544289 2017 raft_consensus.cc:729] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fcad2ad6cf46b7b3798fdbcc8fd2cc, State: Initialized, Role: FOLLOWER
I20260928 08:37:11.544445 2018 raft_consensus.cc:348] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } } peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } }
I20260928 08:37:11.544813 2018 raft_consensus.cc:374] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:11.544911 2018 raft_consensus.cc:729] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0af96e414c403faeaf394b27744a76, State: Initialized, Role: FOLLOWER
I20260928 08:37:11.545058 2017 consensus_queue.cc:260] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } } peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } }
I20260928 08:37:11.545564 2018 consensus_queue.cc:260] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } } peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } }
I20260928 08:37:11.546010 2019 raft_consensus.cc:348] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } } peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } }
I20260928 08:37:11.546396 2019 raft_consensus.cc:374] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:11.546519 2019 raft_consensus.cc:729] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 143aa94c7705441c9dbd0f56d21f56b9, State: Initialized, Role: FOLLOWER
I20260928 08:37:11.547039 2019 consensus_queue.cc:260] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } } peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } }
I20260928 08:37:11.547699 2017 ts_tablet_manager.cc:1428] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260928 08:37:11.548895 1998 heartbeater.cc:503] Master 127.0.97.62:33851 was elected leader, sending a full tablet report...
I20260928 08:37:11.549454 1859 heartbeater.cc:503] Master 127.0.97.62:33851 was elected leader, sending a full tablet report...
I20260928 08:37:11.550119 2019 ts_tablet_manager.cc:1428] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260928 08:37:11.551172 2018 ts_tablet_manager.cc:1428] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76: Time spent starting tablet: real 0.018s user 0.014s sys 0.003s
W20260928 08:37:11.637815 1721 tablet.cc:2369] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:37:11.643971 1860 tablet.cc:2369] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:37:11.664917 1999 tablet.cc:2369] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:37:11.763790 2023 raft_consensus.cc:482] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:37:11.764132 2023 raft_consensus.cc:504] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } } peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } }
I20260928 08:37:11.765733 2023 leader_election.cc:288] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 143aa94c7705441c9dbd0f56d21f56b9 (127.0.97.3:36413), 7a0af96e414c403faeaf394b27744a76 (127.0.97.2:43583)
I20260928 08:37:11.773448 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:11.767153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43583 (local address 127.0.97.1:37415)
0928 08:37:11.767772 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:11.767786 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:11.767886 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:11.768191 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:11.768193 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:11.768209 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:11.768605 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:11.768617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.769699 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.769702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:11.771160 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:11.771173 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.772740 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.772745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:11.772862 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:11.772884 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 08:37:11.772996 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 08:37:11.773111 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":351,"threads_started":1}
I20260928 08:37:11.773476 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:11.767470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37415 (local address 127.0.97.2:43583)
0928 08:37:11.767660 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:11.767667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:11.767874 (+ 207us) server_negotiation.cc:407] Connection header received
0928 08:37:11.767993 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:11.767998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:11.768074 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:11.768175 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:11.768763 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.769560 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.771311 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.772135 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.772670 (+ 535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:11.772769 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:11.773034 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 08:37:11.773150 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:37:11.773526 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:11.766322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53999 (local address 127.0.97.3:36413)
0928 08:37:11.766547 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:11.766556 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:11.767153 (+ 597us) server_negotiation.cc:407] Connection header received
0928 08:37:11.767417 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:11.767423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:11.767506 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:11.767613 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:11.768324 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.769210 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.771159 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.771985 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.772080 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:11.772155 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:11.772979 (+ 824us) server_negotiation.cc:299] Negotiation successful
0928 08:37:11.773109 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:37:11.773597 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:11.766222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36413 (local address 127.0.97.1:53999)
0928 08:37:11.767037 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:11.767058 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:11.767199 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:11.767755 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:11.767765 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:11.767799 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:11.768164 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:11.768176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.769333 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.769338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:11.771011 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:11.771022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.772643 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.772650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:11.772825 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:11.772849 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 08:37:11.772925 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 08:37:11.773063 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":461,"threads_started":1}
I20260928 08:37:11.774725 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5718edda114106ba2fa2e433b73e44" candidate_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0af96e414c403faeaf394b27744a76" is_pre_election: true
I20260928 08:37:11.774725 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5718edda114106ba2fa2e433b73e44" candidate_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143aa94c7705441c9dbd0f56d21f56b9" is_pre_election: true
I20260928 08:37:11.775216 1953 raft_consensus.cc:2454] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fcad2ad6cf46b7b3798fdbcc8fd2cc in term 0.
I20260928 08:37:11.775267 1814 raft_consensus.cc:2454] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fcad2ad6cf46b7b3798fdbcc8fd2cc in term 0.
I20260928 08:37:11.775872 1602 leader_election.cc:302] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 143aa94c7705441c9dbd0f56d21f56b9, 80fcad2ad6cf46b7b3798fdbcc8fd2cc; no voters:
I20260928 08:37:11.776227 2023 raft_consensus.cc:2781] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:37:11.776434 2023 raft_consensus.cc:482] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:37:11.776559 2023 raft_consensus.cc:3037] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:11.778759 2023 raft_consensus.cc:504] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } } peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } }
I20260928 08:37:11.779297 2023 leader_election.cc:288] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [CANDIDATE]: Term 1 election: Requested vote from peers 143aa94c7705441c9dbd0f56d21f56b9 (127.0.97.3:36413), 7a0af96e414c403faeaf394b27744a76 (127.0.97.2:43583)
I20260928 08:37:11.779795 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5718edda114106ba2fa2e433b73e44" candidate_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143aa94c7705441c9dbd0f56d21f56b9"
I20260928 08:37:11.779930 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5718edda114106ba2fa2e433b73e44" candidate_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0af96e414c403faeaf394b27744a76"
I20260928 08:37:11.779996 1953 raft_consensus.cc:3037] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:11.780156 1814 raft_consensus.cc:3037] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:11.782128 1953 raft_consensus.cc:2454] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fcad2ad6cf46b7b3798fdbcc8fd2cc in term 1.
I20260928 08:37:11.782347 1814 raft_consensus.cc:2454] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fcad2ad6cf46b7b3798fdbcc8fd2cc in term 1.
I20260928 08:37:11.782549 1602 leader_election.cc:302] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 143aa94c7705441c9dbd0f56d21f56b9, 80fcad2ad6cf46b7b3798fdbcc8fd2cc; no voters:
I20260928 08:37:11.782832 2023 raft_consensus.cc:2781] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:11.783301 2023 raft_consensus.cc:686] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 1 LEADER]: Becoming Leader. State: Replica: 80fcad2ad6cf46b7b3798fdbcc8fd2cc, State: Running, Role: LEADER
I20260928 08:37:11.783764 2023 consensus_queue.cc:237] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } } peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } }
I20260928 08:37:11.788691 1512 catalog_manager.cc:5495] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fcad2ad6cf46b7b3798fdbcc8fd2cc (127.0.97.1). New cstate: current_term: 1 leader_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } health_report { overall_health: UNKNOWN } } }
I20260928 08:37:11.879837 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:11.872250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37472 (local address 127.0.97.1:35691)
0928 08:37:11.872635 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:11.872642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:11.872667 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:37:11.872897 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:11.872904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:11.873047 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:11.873212 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:37:11.873724 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.874667 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.876516 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:11.877172 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:11.877306 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:11.878033 (+ 727us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:37:11.878778 (+ 745us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:37:11.878898 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:11.879393 (+ 495us) server_negotiation.cc:299] Negotiation successful
0928 08:37:11.879505 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260928 08:37:11.888347 1814 raft_consensus.cc:1261] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76 [term 1 FOLLOWER]: Refusing update from remote peer 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:11.888381 1953 raft_consensus.cc:1261] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9 [term 1 FOLLOWER]: Refusing update from remote peer 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:11.889437 2023 consensus_queue.cc:1035] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:11.889786 2028 consensus_queue.cc:1035] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:11.906594 2033 mvcc.cc:205] Tried to move back new op lower bound from 7334234652202848256 to 7334234651791659008. Current Snapshot: MvccSnapshot[applied={T|T < 7334234652202848256}]
I20260928 08:37:11.923441 1512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:11.923743 1512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:11.927049 1512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279: Sending DeleteTablet for 3 replicas of tablet 3f5718edda114106ba2fa2e433b73e44
I20260928 08:37:11.928193 1650 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f5718edda114106ba2fa2e433b73e44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37456
I20260928 08:37:11.928296 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f5718edda114106ba2fa2e433b73e44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39398
I20260928 08:37:11.928447 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f5718edda114106ba2fa2e433b73e44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42310
I20260928 08:37:11.934666 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:37:11.936652 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:37:11.944975 2042 tablet_replica.cc:330] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9: stopping tablet replica
I20260928 08:37:11.945799 2042 raft_consensus.cc:2229] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:11.948167 1649 tablet_service.cc:1467] Processing CreateTablet for tablet b96a625b33e0465fa18e93065d6b56d1 (DEFAULT_TABLE table=test-table [id=7a47d65ba6284e6ebbd527467db116b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:11.948969 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96a625b33e0465fa18e93065d6b56d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:11.949574 2041 tablet_replica.cc:330] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76: stopping tablet replica
I20260928 08:37:11.949991 2041 raft_consensus.cc:2229] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:11.950495 2041 raft_consensus.cc:2258] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:11.951794 1788 tablet_service.cc:1467] Processing CreateTablet for tablet b96a625b33e0465fa18e93065d6b56d1 (DEFAULT_TABLE table=test-table [id=7a47d65ba6284e6ebbd527467db116b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:11.952239 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96a625b33e0465fa18e93065d6b56d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:11.958875 1928 tablet_service.cc:1467] Processing CreateTablet for tablet b96a625b33e0465fa18e93065d6b56d1 (DEFAULT_TABLE table=test-table [id=7a47d65ba6284e6ebbd527467db116b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:11.959367 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96a625b33e0465fa18e93065d6b56d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:11.959748 2017 tablet_bootstrap.cc:492] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Bootstrap starting.
I20260928 08:37:11.962080 2017 tablet_bootstrap.cc:654] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:11.963236 2018 tablet_bootstrap.cc:492] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76: Bootstrap starting.
I20260928 08:37:11.965327 2018 tablet_bootstrap.cc:654] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:11.966935 2042 raft_consensus.cc:2258] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:11.968752 2019 tablet_bootstrap.cc:492] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9: Bootstrap starting.
I20260928 08:37:11.969134 2040 tablet_replica.cc:330] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: stopping tablet replica
I20260928 08:37:11.969575 2040 raft_consensus.cc:2229] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:37:11.970034 2019 tablet_bootstrap.cc:654] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:11.970199 2040 raft_consensus.cc:2258] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:11.972240 2041 ts_tablet_manager.cc:1905] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:11.972527 2019 tablet_bootstrap.cc:492] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9: No bootstrap required, opened a new log
I20260928 08:37:11.972726 2019 ts_tablet_manager.cc:1397] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 08:37:11.973709 2019 raft_consensus.cc:348] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:11.974076 2019 raft_consensus.cc:374] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:11.974223 2019 raft_consensus.cc:729] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 143aa94c7705441c9dbd0f56d21f56b9, State: Initialized, Role: FOLLOWER
I20260928 08:37:11.974632 2019 consensus_queue.cc:260] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:11.975294 2019 ts_tablet_manager.cc:1428] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:37:11.976596 2040 ts_tablet_manager.cc:1905] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:11.984570 2040 ts_tablet_manager.cc:1918] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:37:11.984606 2042 ts_tablet_manager.cc:1905] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:11.984752 2040 log.cc:1196] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-0/wal/wals/3f5718edda114106ba2fa2e433b73e44
I20260928 08:37:11.985306 2040 ts_tablet_manager.cc:1939] T 3f5718edda114106ba2fa2e433b73e44 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Deleting consensus metadata
I20260928 08:37:11.988485 2017 tablet_bootstrap.cc:492] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: No bootstrap required, opened a new log
I20260928 08:37:11.989086 2017 ts_tablet_manager.cc:1397] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.001s
I20260928 08:37:11.990887 2041 ts_tablet_manager.cc:1918] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:37:11.991056 2041 log.cc:1196] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/3f5718edda114106ba2fa2e433b73e44
I20260928 08:37:11.991097 2017 raft_consensus.cc:348] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:11.991338 2017 raft_consensus.cc:374] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:11.991451 2017 raft_consensus.cc:729] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fcad2ad6cf46b7b3798fdbcc8fd2cc, State: Initialized, Role: FOLLOWER
I20260928 08:37:11.991612 2041 ts_tablet_manager.cc:1939] T 3f5718edda114106ba2fa2e433b73e44 P 7a0af96e414c403faeaf394b27744a76: Deleting consensus metadata
I20260928 08:37:11.992743 2018 tablet_bootstrap.cc:492] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76: No bootstrap required, opened a new log
I20260928 08:37:11.992895 2018 ts_tablet_manager.cc:1397] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.003s
I20260928 08:37:11.993530 1499 catalog_manager.cc:4841] TS 7a0af96e414c403faeaf394b27744a76 (127.0.97.2:43583): tablet 3f5718edda114106ba2fa2e433b73e44 (table test-table [id=b15d480394d448c3b7fa0baf40a54ba0]) successfully deleted
I20260928 08:37:11.993607 2042 ts_tablet_manager.cc:1918] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:37:11.993718 2018 raft_consensus.cc:348] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:11.993882 2042 log.cc:1196] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/3f5718edda114106ba2fa2e433b73e44
I20260928 08:37:11.994524 2042 ts_tablet_manager.cc:1939] T 3f5718edda114106ba2fa2e433b73e44 P 143aa94c7705441c9dbd0f56d21f56b9: Deleting consensus metadata
I20260928 08:37:11.994881 1500 catalog_manager.cc:4841] TS 80fcad2ad6cf46b7b3798fdbcc8fd2cc (127.0.97.1:35691): tablet 3f5718edda114106ba2fa2e433b73e44 (table test-table [id=b15d480394d448c3b7fa0baf40a54ba0]) successfully deleted
I20260928 08:37:11.995126 2018 raft_consensus.cc:374] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:11.995245 2018 raft_consensus.cc:729] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0af96e414c403faeaf394b27744a76, State: Initialized, Role: FOLLOWER
I20260928 08:37:11.995373 2017 consensus_queue.cc:260] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:11.995523 2018 consensus_queue.cc:260] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:11.996193 2018 ts_tablet_manager.cc:1428] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 08:37:11.997053 1499 catalog_manager.cc:4841] TS 143aa94c7705441c9dbd0f56d21f56b9 (127.0.97.3:36413): tablet 3f5718edda114106ba2fa2e433b73e44 (table test-table [id=b15d480394d448c3b7fa0baf40a54ba0]) successfully deleted
I20260928 08:37:11.997205 2017 ts_tablet_manager.cc:1428] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20260928 08:37:12.246876 2028 raft_consensus.cc:482] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:37:12.247231 2028 raft_consensus.cc:504] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:12.247926 2028 leader_election.cc:288] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a0af96e414c403faeaf394b27744a76 (127.0.97.2:43583), 143aa94c7705441c9dbd0f56d21f56b9 (127.0.97.3:36413)
I20260928 08:37:12.248595 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96a625b33e0465fa18e93065d6b56d1" candidate_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0af96e414c403faeaf394b27744a76" is_pre_election: true
I20260928 08:37:12.248685 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96a625b33e0465fa18e93065d6b56d1" candidate_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143aa94c7705441c9dbd0f56d21f56b9" is_pre_election: true
I20260928 08:37:12.248852 1814 raft_consensus.cc:2454] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fcad2ad6cf46b7b3798fdbcc8fd2cc in term 0.
I20260928 08:37:12.248901 1953 raft_consensus.cc:2454] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80fcad2ad6cf46b7b3798fdbcc8fd2cc in term 0.
I20260928 08:37:12.249341 1602 leader_election.cc:302] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a0af96e414c403faeaf394b27744a76, 80fcad2ad6cf46b7b3798fdbcc8fd2cc; no voters:
I20260928 08:37:12.249651 2028 raft_consensus.cc:2781] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:37:12.249797 2028 raft_consensus.cc:482] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:37:12.249904 2028 raft_consensus.cc:3037] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:12.253299 2028 raft_consensus.cc:504] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:12.254065 2028 leader_election.cc:288] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [CANDIDATE]: Term 1 election: Requested vote from peers 7a0af96e414c403faeaf394b27744a76 (127.0.97.2:43583), 143aa94c7705441c9dbd0f56d21f56b9 (127.0.97.3:36413)
I20260928 08:37:12.254560 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96a625b33e0465fa18e93065d6b56d1" candidate_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0af96e414c403faeaf394b27744a76"
I20260928 08:37:12.254627 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96a625b33e0465fa18e93065d6b56d1" candidate_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143aa94c7705441c9dbd0f56d21f56b9"
I20260928 08:37:12.254796 1814 raft_consensus.cc:3037] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:12.254853 1953 raft_consensus.cc:3037] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:12.257273 1953 raft_consensus.cc:2454] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fcad2ad6cf46b7b3798fdbcc8fd2cc in term 1.
I20260928 08:37:12.257706 1602 leader_election.cc:302] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 143aa94c7705441c9dbd0f56d21f56b9, 80fcad2ad6cf46b7b3798fdbcc8fd2cc; no voters:
I20260928 08:37:12.257997 2028 raft_consensus.cc:2781] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:12.258194 2028 raft_consensus.cc:686] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 1 LEADER]: Becoming Leader. State: Replica: 80fcad2ad6cf46b7b3798fdbcc8fd2cc, State: Running, Role: LEADER
I20260928 08:37:12.258249 1814 raft_consensus.cc:2454] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80fcad2ad6cf46b7b3798fdbcc8fd2cc in term 1.
I20260928 08:37:12.258445 2028 consensus_queue.cc:237] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:12.261447 1512 catalog_manager.cc:5495] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 80fcad2ad6cf46b7b3798fdbcc8fd2cc (127.0.97.1). New cstate: current_term: 1 leader_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } health_report { overall_health: UNKNOWN } } }
Sep 28 08:37:12 dist-test-slave-w6c0 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:12.325250 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.309027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33851 (local address 127.0.0.1:38242)
0928 08:37:12.310172 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:12.310194 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:12.310337 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:12.310779 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:12.310782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:12.311416 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:12.311609 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:12.311617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.313155 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.313159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:12.314126 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:12.314137 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.315091 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.315096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:12.315234 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.315946 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:12.315962 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:12.318562 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:12.322419 (+ 3857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:12.322429 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:12.322435 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:12.323229 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:12.323614 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:12.323617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:12.323619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:12.323766 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:12.324455 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:12.324461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:12.324602 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 08:37:12.324740 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 08:37:12.324962 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"thread_start_us":924,"threads_started":1}
I20260928 08:37:12.325495 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.309075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.0.97.62:33851)
0928 08:37:12.309772 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:12.309782 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:12.310311 (+ 529us) server_negotiation.cc:407] Connection header received
0928 08:37:12.310462 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:12.310468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:12.310580 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:12.310709 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:12.311787 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.312998 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.314275 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.314924 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.315038 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.318750 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:12.318757 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:12.318762 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:12.318826 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:12.322269 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:12.323348 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:12.323355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:12.323359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:12.323504 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:12.323868 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:12.323872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:12.323877 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:12.324208 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:12.324471 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:12.324881 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 08:37:12.325095 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":321,"threads_started":1}
I20260928 08:37:12.331712 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38242:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 08:37:12.332787 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:37:12.339614 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 66b0badb72d9474f9bc56d8d1f0b72c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e5f5d055d194a938646e2cf914ab5d4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:37:12.339926 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 66b0badb72d9474f9bc56d8d1f0b72c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e5f5d055d194a938646e2cf914ab5d4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:37:12.340495 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b0badb72d9474f9bc56d8d1f0b72c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:12.340603 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b0badb72d9474f9bc56d8d1f0b72c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:12.343662 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 66b0badb72d9474f9bc56d8d1f0b72c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e5f5d055d194a938646e2cf914ab5d4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:37:12.344250 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b0badb72d9474f9bc56d8d1f0b72c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:12.347893 2018 tablet_bootstrap.cc:492] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76: Bootstrap starting.
I20260928 08:37:12.348925 2017 tablet_bootstrap.cc:492] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Bootstrap starting.
I20260928 08:37:12.349610 2018 tablet_bootstrap.cc:654] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:12.350745 2017 tablet_bootstrap.cc:654] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:12.352834 2019 tablet_bootstrap.cc:492] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9: Bootstrap starting.
I20260928 08:37:12.354576 2019 tablet_bootstrap.cc:654] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:12.361904 2018 tablet_bootstrap.cc:492] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76: No bootstrap required, opened a new log
I20260928 08:37:12.362128 2018 ts_tablet_manager.cc:1397] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20260928 08:37:12.363035 2018 raft_consensus.cc:348] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:12.363309 2018 raft_consensus.cc:374] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:12.363404 2018 raft_consensus.cc:729] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0af96e414c403faeaf394b27744a76, State: Initialized, Role: FOLLOWER
I20260928 08:37:12.363706 2018 consensus_queue.cc:260] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:12.364303 2018 ts_tablet_manager.cc:1428] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 08:37:12.364233 2024 tablet_replica.cc:439] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } } }
I20260928 08:37:12.364612 2024 tablet_replica.cc:442] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:37:12.364641 2019 tablet_bootstrap.cc:492] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9: No bootstrap required, opened a new log
I20260928 08:37:12.364858 2019 ts_tablet_manager.cc:1397] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260928 08:37:12.365617 2019 raft_consensus.cc:348] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:12.365895 2019 raft_consensus.cc:374] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:12.366010 2019 raft_consensus.cc:729] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 143aa94c7705441c9dbd0f56d21f56b9, State: Initialized, Role: FOLLOWER
I20260928 08:37:12.366164 2017 tablet_bootstrap.cc:492] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: No bootstrap required, opened a new log
I20260928 08:37:12.366367 2017 ts_tablet_manager.cc:1397] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.008s
I20260928 08:37:12.366324 2019 consensus_queue.cc:260] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:12.367048 2019 ts_tablet_manager.cc:1428] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:37:12.366943 2025 tablet_replica.cc:439] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } } }
I20260928 08:37:12.367318 2025 tablet_replica.cc:442] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:37:12.367509 2017 raft_consensus.cc:348] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:12.367760 2017 raft_consensus.cc:374] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:12.368414 2017 raft_consensus.cc:729] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fcad2ad6cf46b7b3798fdbcc8fd2cc, State: Initialized, Role: FOLLOWER
I20260928 08:37:12.368742 2017 consensus_queue.cc:260] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:12.369319 2028 tablet_replica.cc:439] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } } }
I20260928 08:37:12.369585 2028 tablet_replica.cc:442] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:37:12.370103 2017 ts_tablet_manager.cc:1428] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 08:37:12.372041 2024 raft_consensus.cc:482] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:37:12.372236 2024 raft_consensus.cc:504] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:12.373948 2024 leader_election.cc:288] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fcad2ad6cf46b7b3798fdbcc8fd2cc (127.0.97.1:35691), 143aa94c7705441c9dbd0f56d21f56b9 (127.0.97.3:36413)
I20260928 08:37:12.380556 2025 raft_consensus.cc:482] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:37:12.380792 2025 raft_consensus.cc:504] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:12.382937 2025 leader_election.cc:288] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80fcad2ad6cf46b7b3798fdbcc8fd2cc (127.0.97.1:35691), 7a0af96e414c403faeaf394b27744a76 (127.0.97.2:43583)
I20260928 08:37:12.397537 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.383414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35691 (local address 127.0.97.3:59031)
0928 08:37:12.387065 (+ 3651us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:12.387092 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:12.387281 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:12.391799 (+ 4518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:12.391804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:12.391833 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:12.392110 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:12.392119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.393302 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.393316 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:12.395211 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:12.395223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.396607 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.396617 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:12.396863 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.396913 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 08:37:12.397000 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 08:37:12.397162 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3464,"threads_started":1}
I20260928 08:37:12.398548 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.384663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59031 (local address 127.0.97.1:35691)
0928 08:37:12.391022 (+ 6359us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:12.391028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:12.391055 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:37:12.391145 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:12.391151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:12.391224 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:12.391782 (+ 558us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:12.392289 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.393142 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.395368 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.396203 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.396365 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.396519 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:12.397957 (+ 1438us) server_negotiation.cc:299] Negotiation successful
0928 08:37:12.398111 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6187,"thread_start_us":6019,"threads_started":1}
I20260928 08:37:12.401999 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b0badb72d9474f9bc56d8d1f0b72c6" candidate_uuid: "143aa94c7705441c9dbd0f56d21f56b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" is_pre_election: true
I20260928 08:37:12.402441 1675 raft_consensus.cc:2454] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 143aa94c7705441c9dbd0f56d21f56b9 in term 0.
I20260928 08:37:12.402868 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.374573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35409 (local address 127.0.97.1:35691)
0928 08:37:12.374784 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:12.374794 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:12.379267 (+ 4473us) server_negotiation.cc:407] Connection header received
0928 08:37:12.379979 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:12.379988 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:12.380091 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:12.380166 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:12.381099 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.382208 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.400326 (+ 18118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.401679 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.401819 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.401921 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:12.402429 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 08:37:12.402610 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 08:37:12.403376 1882 leader_election.cc:302] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 143aa94c7705441c9dbd0f56d21f56b9, 80fcad2ad6cf46b7b3798fdbcc8fd2cc; no voters:
I20260928 08:37:12.403517 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.374131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35691 (local address 127.0.97.2:35409)
0928 08:37:12.379145 (+ 5014us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:12.379172 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:12.379770 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:12.380445 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:12.380449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:12.380478 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:12.380909 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:12.380924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.382393 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.382398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:12.398852 (+ 16454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:12.398867 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.402057 (+ 3190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.402062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:12.402247 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.402278 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 08:37:12.403061 (+ 783us) client_negotiation.cc:240] Negotiation successful
0928 08:37:12.403204 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4861,"thread_start_us":4728,"threads_started":1}
I20260928 08:37:12.403962 2025 raft_consensus.cc:2781] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:37:12.404726 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b0badb72d9474f9bc56d8d1f0b72c6" candidate_uuid: "7a0af96e414c403faeaf394b27744a76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" is_pre_election: true
I20260928 08:37:12.404917 1675 raft_consensus.cc:2454] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a0af96e414c403faeaf394b27744a76 in term 0.
I20260928 08:37:12.404956 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.374142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36413 (local address 127.0.97.2:44371)
0928 08:37:12.396135 (+ 21993us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:12.396166 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:12.396327 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:12.397321 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:12.397326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:12.397367 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:12.397822 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:12.397840 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.399552 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.399560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:12.401896 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:12.401911 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.404032 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.404043 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:12.404243 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.404286 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 08:37:12.404392 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 08:37:12.404526 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21840,"mutex_wait_us":356,"thread_start_us":21264,"threads_started":1}
I20260928 08:37:12.405573 1743 leader_election.cc:302] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a0af96e414c403faeaf394b27744a76, 80fcad2ad6cf46b7b3798fdbcc8fd2cc; no voters:
I20260928 08:37:12.405896 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.374482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44371 (local address 127.0.97.3:36413)
0928 08:37:12.389627 (+ 15145us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:12.389634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:12.396284 (+ 6650us) server_negotiation.cc:407] Connection header received
0928 08:37:12.396468 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:12.396476 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:12.396556 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:12.396654 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:12.398054 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.399387 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.402090 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.403577 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.403715 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.403872 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:12.405367 (+ 1495us) server_negotiation.cc:299] Negotiation successful
0928 08:37:12.405519 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":14671,"thread_start_us":14988,"threads_started":1}
I20260928 08:37:12.405995 2024 raft_consensus.cc:2781] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:37:12.406117 2024 raft_consensus.cc:482] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:37:12.406121 2025 raft_consensus.cc:482] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:37:12.406245 2025 raft_consensus.cc:3037] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:12.406227 2024 raft_consensus.cc:3037] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:12.408831 2024 raft_consensus.cc:504] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:12.409194 2025 raft_consensus.cc:504] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:12.409566 2024 leader_election.cc:288] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 [CANDIDATE]: Term 1 election: Requested vote from peers 80fcad2ad6cf46b7b3798fdbcc8fd2cc (127.0.97.1:35691), 143aa94c7705441c9dbd0f56d21f56b9 (127.0.97.3:36413)
I20260928 08:37:12.409989 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b0badb72d9474f9bc56d8d1f0b72c6" candidate_uuid: "7a0af96e414c403faeaf394b27744a76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc"
I20260928 08:37:12.410174 1675 raft_consensus.cc:3037] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:12.411051 2025 leader_election.cc:288] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9 [CANDIDATE]: Term 1 election: Requested vote from peers 80fcad2ad6cf46b7b3798fdbcc8fd2cc (127.0.97.1:35691), 7a0af96e414c403faeaf394b27744a76 (127.0.97.2:43583)
I20260928 08:37:12.411250 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b0badb72d9474f9bc56d8d1f0b72c6" candidate_uuid: "7a0af96e414c403faeaf394b27744a76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143aa94c7705441c9dbd0f56d21f56b9" is_pre_election: true
I20260928 08:37:12.411577 1953 raft_consensus.cc:2379] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a0af96e414c403faeaf394b27744a76 in current term 1: Already voted for candidate 143aa94c7705441c9dbd0f56d21f56b9 in this term.
I20260928 08:37:12.411852 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b0badb72d9474f9bc56d8d1f0b72c6" candidate_uuid: "143aa94c7705441c9dbd0f56d21f56b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc"
I20260928 08:37:12.411971 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b0badb72d9474f9bc56d8d1f0b72c6" candidate_uuid: "7a0af96e414c403faeaf394b27744a76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143aa94c7705441c9dbd0f56d21f56b9"
I20260928 08:37:12.412293 1952 raft_consensus.cc:2379] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a0af96e414c403faeaf394b27744a76 in current term 1: Already voted for candidate 143aa94c7705441c9dbd0f56d21f56b9 in this term.
I20260928 08:37:12.413240 1675 raft_consensus.cc:2454] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0af96e414c403faeaf394b27744a76 in term 1.
I20260928 08:37:12.413580 1743 leader_election.cc:302] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a0af96e414c403faeaf394b27744a76, 80fcad2ad6cf46b7b3798fdbcc8fd2cc; no voters: 143aa94c7705441c9dbd0f56d21f56b9
I20260928 08:37:12.414324 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.384393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41699 (local address 127.0.97.2:43583)
0928 08:37:12.391670 (+ 7277us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:12.391674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:12.397871 (+ 6197us) server_negotiation.cc:407] Connection header received
0928 08:37:12.398947 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:12.398953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:12.399018 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:12.399086 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:12.399847 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.401297 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.403856 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.410391 (+ 6535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.410473 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.410661 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:12.413777 (+ 3116us) server_negotiation.cc:299] Negotiation successful
0928 08:37:12.413935 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7179,"thread_start_us":7072,"threads_started":1}
I20260928 08:37:12.414587 2024 raft_consensus.cc:2781] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:12.415081 2024 raft_consensus.cc:686] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 [term 1 LEADER]: Becoming Leader. State: Replica: 7a0af96e414c403faeaf394b27744a76, State: Running, Role: LEADER
I20260928 08:37:12.415480 2024 consensus_queue.cc:237] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } }
I20260928 08:37:12.417569 2064 tablet_replica.cc:439] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76: TxnStatusTablet state changed. Reason: New leader 7a0af96e414c403faeaf394b27744a76. Latest consensus state: current_term: 1 leader_uuid: "7a0af96e414c403faeaf394b27744a76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } } }
I20260928 08:37:12.418421 2064 tablet_replica.cc:442] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:37:12.420562 1513 catalog_manager.cc:5495] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a0af96e414c403faeaf394b27744a76 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "7a0af96e414c403faeaf394b27744a76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } health_report { overall_health: UNKNOWN } } }
I20260928 08:37:12.423103 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.383935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43583 (local address 127.0.97.3:41699)
0928 08:37:12.397765 (+ 13830us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:12.397786 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:12.398776 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:12.399348 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:12.399353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:12.399377 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:12.399621 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:12.399636 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.401486 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.401492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:12.403646 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:12.403662 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.413284 (+ 9622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.413289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:12.413544 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.413588 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 08:37:12.422647 (+ 9059us) client_negotiation.cc:240] Negotiation successful
0928 08:37:12.422813 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13712,"mutex_wait_us":5,"thread_start_us":2760,"threads_started":1}
I20260928 08:37:12.423794 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b0badb72d9474f9bc56d8d1f0b72c6" candidate_uuid: "143aa94c7705441c9dbd0f56d21f56b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0af96e414c403faeaf394b27744a76" is_pre_election: true
I20260928 08:37:12.424285 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b0badb72d9474f9bc56d8d1f0b72c6" candidate_uuid: "143aa94c7705441c9dbd0f56d21f56b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0af96e414c403faeaf394b27744a76"
I20260928 08:37:12.425746 1881 leader_election.cc:302] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 143aa94c7705441c9dbd0f56d21f56b9; no voters: 7a0af96e414c403faeaf394b27744a76, 80fcad2ad6cf46b7b3798fdbcc8fd2cc
I20260928 08:37:12.426366 2025 raft_consensus.cc:2726] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 08:37:12.430828 2066 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 08:37:12.431068 2066 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 08:37:12.451449 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.444688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43583 (local address 127.0.0.1:39404)
0928 08:37:12.444914 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:12.444941 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:12.445071 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:12.445597 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:12.445602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:12.445617 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:37:12.445817 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:12.445828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.447500 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.447505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:12.448844 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:12.448855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.449728 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.449734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:12.449923 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.449978 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:37:12.450892 (+ 914us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:37:12.450915 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 08:37:12.451001 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 08:37:12.451151 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 08:37:12.451421 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.444706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39404 (local address 127.0.97.2:43583)
0928 08:37:12.444940 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:12.444947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:12.445003 (+ 56us) server_negotiation.cc:407] Connection header received
0928 08:37:12.445274 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:12.445281 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:12.445403 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:12.445568 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:37:12.445982 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.447331 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.449000 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.449557 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.449694 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.450187 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:37:12.450781 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:37:12.450940 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:12.451023 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 08:37:12.451109 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:37:12.456722 1952 raft_consensus.cc:1261] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9 [term 1 FOLLOWER]: Refusing update from remote peer 7a0af96e414c403faeaf394b27744a76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:12.456903 1675 raft_consensus.cc:1261] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 1 FOLLOWER]: Refusing update from remote peer 7a0af96e414c403faeaf394b27744a76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:12.457479 2064 consensus_queue.cc:1035] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:12.458280 2064 consensus_queue.cc:1035] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:12.463559 2028 tablet_replica.cc:439] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: TxnStatusTablet state changed. Reason: New leader 7a0af96e414c403faeaf394b27744a76. Latest consensus state: current_term: 1 leader_uuid: "7a0af96e414c403faeaf394b27744a76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } } }
I20260928 08:37:12.463892 2028 tablet_replica.cc:442] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:37:12.465337 2025 tablet_replica.cc:439] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9: TxnStatusTablet state changed. Reason: New leader 7a0af96e414c403faeaf394b27744a76. Latest consensus state: current_term: 1 leader_uuid: "7a0af96e414c403faeaf394b27744a76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } } }
I20260928 08:37:12.465556 2025 tablet_replica.cc:442] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:37:12.469374 2025 tablet_replica.cc:439] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a0af96e414c403faeaf394b27744a76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } } }
I20260928 08:37:12.469609 2025 tablet_replica.cc:442] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 143aa94c7705441c9dbd0f56d21f56b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:37:12.470291 2072 mvcc.cc:205] Tried to move back new op lower bound from 7334234654534541312 to 7334234654377857024. Current Snapshot: MvccSnapshot[applied={T|T < 7334234654534541312}]
I20260928 08:37:12.474018 2028 tablet_replica.cc:439] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a0af96e414c403faeaf394b27744a76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } } }
I20260928 08:37:12.474306 2028 tablet_replica.cc:442] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:37:12.479707 2024 tablet_replica.cc:439] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a0af96e414c403faeaf394b27744a76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } } }
I20260928 08:37:12.480319 2024 tablet_replica.cc:442] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:37:12.480636 2064 tablet_replica.cc:439] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a0af96e414c403faeaf394b27744a76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80fcad2ad6cf46b7b3798fdbcc8fd2cc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35691 } } peers { permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 } } peers { permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 } } }
I20260928 08:37:12.480867 2064 tablet_replica.cc:442] T 66b0badb72d9474f9bc56d8d1f0b72c6 P 7a0af96e414c403faeaf394b27744a76: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:37:12.501435 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.495986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37609 (local address 127.0.97.2:43583)
0928 08:37:12.496187 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:12.496196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:12.496614 (+ 418us) server_negotiation.cc:407] Connection header received
0928 08:37:12.496845 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:12.496849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:12.496939 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:12.497046 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:37:12.497505 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.498329 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.499680 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.500091 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.500247 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.500680 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:37:12.500789 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:37:12.500882 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:12.501026 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 08:37:12.501105 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 08:37:12.501485 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.495921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43583 (local address 127.0.97.1:37609)
0928 08:37:12.496495 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:12.496512 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:12.496685 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:12.497061 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:12.497065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:12.497077 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:37:12.497372 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:12.497381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.498463 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.498471 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:12.499530 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:12.499542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.500288 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.500297 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:12.500453 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.500507 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:37:12.500897 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:37:12.500909 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 08:37:12.501003 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 08:37:12.501148 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":294,"threads_started":1}
I20260928 08:37:12.510576 1952 raft_consensus.cc:1261] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9 [term 1 FOLLOWER]: Refusing update from remote peer 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:12.510931 1809 raft_consensus.cc:1261] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76 [term 1 FOLLOWER]: Refusing update from remote peer 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:12.511168 2023 consensus_queue.cc:1035] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "143aa94c7705441c9dbd0f56d21f56b9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:12.511445 2028 consensus_queue.cc:1035] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0af96e414c403faeaf394b27744a76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:12.519791 2069 mvcc.cc:205] Tried to move back new op lower bound from 7334234654757859328 to 7334234653733732352. Current Snapshot: MvccSnapshot[applied={T|T < 7334234654757859328}]
I20260928 08:37:12.552400 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.544845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42347 (local address 127.0.97.1:35691)
0928 08:37:12.545087 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:12.545093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:12.546394 (+ 1301us) server_negotiation.cc:407] Connection header received
0928 08:37:12.546858 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:12.546865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:12.546952 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:12.547053 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:37:12.547515 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.548631 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.550564 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.551081 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.551154 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.551669 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:37:12.551802 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:37:12.551900 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:12.551992 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 08:37:12.552078 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:37:12.552640 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.544954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35691 (local address 127.0.97.2:42347)
0928 08:37:12.546282 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:12.546317 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:12.546706 (+ 389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:12.547061 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:12.547063 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:12.547072 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:37:12.547369 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:12.547377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.548759 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.548762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:12.550386 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:12.550403 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.551315 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.551319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:12.551451 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.551501 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:37:12.551904 (+ 403us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:37:12.551911 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 08:37:12.551969 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 08:37:12.552145 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1131,"thread_start_us":1180,"threads_started":1}
I20260928 08:37:12.584923 1513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:12.585193 1513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:12.588380 1513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P adcbde99d7854b5eb1aa9d6bb8dd1279: Sending DeleteTablet for 3 replicas of tablet b96a625b33e0465fa18e93065d6b56d1
I20260928 08:37:12.589304 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet b96a625b33e0465fa18e93065d6b56d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37456
I20260928 08:37:12.589320 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet b96a625b33e0465fa18e93065d6b56d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42310
I20260928 08:37:12.591004 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet b96a625b33e0465fa18e93065d6b56d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39398
I20260928 08:37:12.599649 2091 tablet_replica.cc:330] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76: stopping tablet replica
I20260928 08:37:12.600590 2091 raft_consensus.cc:2229] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:12.601351 2091 raft_consensus.cc:2258] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:12.603487 2091 ts_tablet_manager.cc:1905] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:12.605681 2089 tablet_replica.cc:330] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9: stopping tablet replica
I20260928 08:37:12.606066 2089 raft_consensus.cc:2229] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:12.606410 2089 raft_consensus.cc:2258] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:12.606642 2088 tablet_replica.cc:330] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: stopping tablet replica
I20260928 08:37:12.606842 2088 raft_consensus.cc:2229] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:37:12.607930 2089 ts_tablet_manager.cc:1905] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:12.610888 2088 raft_consensus.cc:2258] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:12.611965 2088 ts_tablet_manager.cc:1905] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:12.614044 2091 ts_tablet_manager.cc:1918] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:37:12.614173 2091 log.cc:1196] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/b96a625b33e0465fa18e93065d6b56d1
I20260928 08:37:12.614686 2091 ts_tablet_manager.cc:1939] T b96a625b33e0465fa18e93065d6b56d1 P 7a0af96e414c403faeaf394b27744a76: Deleting consensus metadata
I20260928 08:37:12.615859 1499 catalog_manager.cc:4841] TS 7a0af96e414c403faeaf394b27744a76 (127.0.97.2:43583): tablet b96a625b33e0465fa18e93065d6b56d1 (table test-table [id=7a47d65ba6284e6ebbd527467db116b9]) successfully deleted
I20260928 08:37:12.622014 2089 ts_tablet_manager.cc:1918] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:37:12.622145 2089 log.cc:1196] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/b96a625b33e0465fa18e93065d6b56d1
I20260928 08:37:12.622673 2089 ts_tablet_manager.cc:1939] T b96a625b33e0465fa18e93065d6b56d1 P 143aa94c7705441c9dbd0f56d21f56b9: Deleting consensus metadata
I20260928 08:37:12.623744 1499 catalog_manager.cc:4841] TS 143aa94c7705441c9dbd0f56d21f56b9 (127.0.97.3:36413): tablet b96a625b33e0465fa18e93065d6b56d1 (table test-table [id=7a47d65ba6284e6ebbd527467db116b9]) successfully deleted
I20260928 08:37:12.624101 2088 ts_tablet_manager.cc:1918] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:37:12.624254 2088 log.cc:1196] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584618551208-388-0/minicluster-data/ts-0/wal/wals/b96a625b33e0465fa18e93065d6b56d1
I20260928 08:37:12.624720 2088 ts_tablet_manager.cc:1939] T b96a625b33e0465fa18e93065d6b56d1 P 80fcad2ad6cf46b7b3798fdbcc8fd2cc: Deleting consensus metadata
I20260928 08:37:12.625581 1500 catalog_manager.cc:4841] TS 80fcad2ad6cf46b7b3798fdbcc8fd2cc (127.0.97.1:35691): tablet b96a625b33e0465fa18e93065d6b56d1 (table test-table [id=7a47d65ba6284e6ebbd527467db116b9]) successfully deleted
Sep 28 08:37:12 dist-test-slave-w6c0 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584631, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:37:12.635949 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.607387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.0.97.1:35691)
0928 08:37:12.607631 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:12.607638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:12.616703 (+ 9065us) server_negotiation.cc:407] Connection header received
0928 08:37:12.616888 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:12.616894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:12.616980 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:12.617104 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:12.619539 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.620697 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.622494 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.623258 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.623381 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.628810 (+ 5429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:12.628827 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:12.628848 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:12.628888 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:12.633110 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:12.633838 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:12.633847 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:12.633851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:12.634004 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:12.634344 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:12.634351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:12.634355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:12.634857 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:12.635140 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:12.635357 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 08:37:12.635575 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260928 08:37:12.636795 1715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37478
I20260928 08:37:12.653823 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.639744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.0.97.62:33851)
0928 08:37:12.640060 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:12.640068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:12.640093 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:37:12.640281 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:12.640284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:12.640358 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:12.640438 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:12.642025 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.642836 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.644652 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.645507 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.645602 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.648392 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:12.648397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:12.648400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:12.648431 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:12.651064 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:12.651788 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:12.651791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:12.651794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:12.651899 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:12.652255 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:12.652259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:12.652262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:12.652592 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:12.652762 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:12.653175 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 08:37:12.653449 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
W20260928 08:37:12.654431 1513 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38256
I20260928 08:37:12.656379 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 1584
I20260928 08:37:12.669612 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.657929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.0.97.1:35691)
0928 08:37:12.658107 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:12.658115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:12.658384 (+ 269us) server_negotiation.cc:407] Connection header received
0928 08:37:12.658590 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:12.658595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:12.658664 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:12.658765 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:12.659779 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.660466 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.661693 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.662088 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.662214 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.664327 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:12.664340 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:12.664344 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:12.664372 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:12.667138 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:12.667748 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:12.667754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:12.667759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:12.667897 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:12.668259 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:12.668267 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:12.668271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:12.668528 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:12.668757 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:12.669047 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 08:37:12.669250 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 08:37:12.670213 1715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37488
W20260928 08:37:12.670781 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 1584: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:37:12.670943 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 1584
I20260928 08:37:12.701056 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 1724
Sep 28 08:37:12 dist-test-slave-w6c0 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584631, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:37:12.722368 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.701932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39410 (local address 127.0.97.2:43583)
0928 08:37:12.702121 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:12.702127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:12.702146 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:37:12.702234 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:12.702239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:12.702310 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:12.702432 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:12.703606 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.704431 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.705709 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.706237 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.706379 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.713180 (+ 6801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:12.713195 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:12.713214 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:12.713280 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:12.718181 (+ 4901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:12.719429 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:12.719438 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:12.719442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:12.719574 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:12.720499 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:12.720504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:12.720507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:12.721053 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:12.721565 (+ 512us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:12.721783 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 08:37:12.722005 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 08:37:12.723405 1854 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39410
W20260928 08:37:12.724246 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 1724: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:37:12.724427 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 1724
I20260928 08:37:12.751869 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 1863
Sep 28 08:37:12 dist-test-slave-w6c0 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584631, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:37:12.767949 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:12.752623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42320 (local address 127.0.97.3:36413)
0928 08:37:12.752835 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:12.752842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:12.752938 (+ 96us) server_negotiation.cc:407] Connection header received
0928 08:37:12.753228 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:12.753234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:12.753310 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:12.753432 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:12.755120 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.756265 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.757682 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:12.758079 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:12.758215 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:12.762497 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:12.762537 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:12.762559 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:12.762604 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:12.765265 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:12.765893 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:12.765902 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:12.765905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:12.766026 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:12.766394 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:12.766400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:12.766404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:12.766824 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:12.767132 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:12.767403 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 08:37:12.767627 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 08:37:12.768646 1993 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42320
W20260928 08:37:12.769339 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 1863: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:37:12.769496 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 1863
I20260928 08:37:12.786262 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 1481
I20260928 08:37:12.787562 1553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:12.900781 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 1481
2026-09-28T08:37:12Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4843 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:37:12 dist-test-slave-w6c0 krb5kdc[2099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:37:12 dist-test-slave-w6c0 krb5kdc[2099](info): set up 2 sockets
Sep 28 08:37:12 dist-test-slave-w6c0 krb5kdc[2099](info): commencing operation
krb5kdc: starting...
W20260928 08:37:13.978909 388 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:37:14 dist-test-slave-w6c0 krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:37:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:37:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:37:14.169318 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46501
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:14.491207 2112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:14.491503 2112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:14.491669 2112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:14.499140 2112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:37:14.499256 2112 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 08:37:14.499352 2112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:14.499436 2112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:14.499527 2112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:37:14.499624 2112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:37:14.509931 2112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46501
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:14.511355 2112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:14.512722 2112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:14.525264 2117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:14.525292 2118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:14.525457 2120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:14.526110 2112 server_base.cc:1034] running on GCE node
I20260928 08:37:14.526968 2112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:14.528439 2112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:14.529667 2112 hybrid_clock.cc:648] HybridClock initialized: now 1790584634529610 us; error 34 us; skew 500 ppm
I20260928 08:37:14.530009 2112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:14 dist-test-slave-w6c0 krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:14.533231 2112 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:37:14.534739 2112 webserver.cc:460] Webserver started at http://127.0.97.62:39541/ using document root <none> and password file <none>
I20260928 08:37:14.535403 2112 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:14.535485 2112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:14.535748 2112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:14.538723 2112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "5f386719bd68486f9689a343758a3bfa"
format_stamp: "Formatted at 2026-09-28 08:37:14 on dist-test-slave-w6c0"
server_key: "2566881217932a107fc8f7c8ce0010a0"
server_key_iv: "fd686e3fe70905ba583cf5372eb88b5a"
server_key_version: "encryptionkey@0"
I20260928 08:37:14.539456 2112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "5f386719bd68486f9689a343758a3bfa"
format_stamp: "Formatted at 2026-09-28 08:37:14 on dist-test-slave-w6c0"
server_key: "2566881217932a107fc8f7c8ce0010a0"
server_key_iv: "fd686e3fe70905ba583cf5372eb88b5a"
server_key_version: "encryptionkey@0"
I20260928 08:37:14.544442 2112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:37:14.547911 2127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:14.549237 2112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:37:14.549422 2112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "5f386719bd68486f9689a343758a3bfa"
format_stamp: "Formatted at 2026-09-28 08:37:14 on dist-test-slave-w6c0"
server_key: "2566881217932a107fc8f7c8ce0010a0"
server_key_iv: "fd686e3fe70905ba583cf5372eb88b5a"
server_key_version: "encryptionkey@0"
I20260928 08:37:14.549641 2112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:14.570740 2112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:14.575222 2112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:14.575503 2112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:14.598320 2112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46501
I20260928 08:37:14.598342 2188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46501 every 8 connection(s)
I20260928 08:37:14.599823 2112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:37:14.603518 2189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:14.607688 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 2112
I20260928 08:37:14.607893 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:37:14.608522 388 external_mini_cluster.cc:1428] Setting key 0f4ca2383db9003a55e2dde2e42a3a8a
I20260928 08:37:14.613988 2189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa: Bootstrap starting.
I20260928 08:37:14.617667 2189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:14.618903 2189 log.cc:824] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:14.622264 2189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa: No bootstrap required, opened a new log
Sep 28 08:37:14 dist-test-slave-w6c0 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:14.629760 2189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f386719bd68486f9689a343758a3bfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46501 } }
I20260928 08:37:14.630152 2189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:14.630270 2189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f386719bd68486f9689a343758a3bfa, State: Initialized, Role: FOLLOWER
I20260928 08:37:14.630869 2189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f386719bd68486f9689a343758a3bfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46501 } }
I20260928 08:37:14.631135 2189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:37:14.631282 2189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:37:14.631459 2189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:14.631669 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:14.610449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.0.97.62:46501)
0928 08:37:14.611328 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:14.611339 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:14.611382 (+ 43us) server_negotiation.cc:407] Connection header received
0928 08:37:14.613077 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:14.613098 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:14.613483 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:14.613801 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:14.615579 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:14.616908 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:14.618731 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:14.619626 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:14.619844 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:14.624185 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:14.624192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:14.624206 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:14.624246 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:14.627384 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:14.628083 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:14.628101 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:14.628108 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:14.628292 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:14.628727 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:14.628738 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:14.628741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:14.629242 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:14.629548 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:14.629974 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 08:37:14.630386 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":378,"threads_started":1}
I20260928 08:37:14.633810 2189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f386719bd68486f9689a343758a3bfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46501 } }
I20260928 08:37:14.634372 2189 leader_election.cc:302] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f386719bd68486f9689a343758a3bfa; no voters:
I20260928 08:37:14.635026 2189 leader_election.cc:288] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:37:14.635191 2194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:14.636216 2194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa [term 1 LEADER]: Becoming Leader. State: Replica: 5f386719bd68486f9689a343758a3bfa, State: Running, Role: LEADER
I20260928 08:37:14.636739 2194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f386719bd68486f9689a343758a3bfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46501 } }
I20260928 08:37:14.637261 2189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:37:14.639313 2195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f386719bd68486f9689a343758a3bfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f386719bd68486f9689a343758a3bfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46501 } } }
I20260928 08:37:14.639622 2195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:14.640447 2201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:37:14.640292 2196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f386719bd68486f9689a343758a3bfa. Latest consensus state: current_term: 1 leader_uuid: "5f386719bd68486f9689a343758a3bfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f386719bd68486f9689a343758a3bfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46501 } } }
I20260928 08:37:14.640621 2196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:14.645633 2201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:37:14.662950 2201 catalog_manager.cc:1347] Generated new cluster ID: 31fc6eab13924064affdc525ca66a40e
I20260928 08:37:14.663090 2201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:37:14.675629 2201 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:37:14.677523 2201 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:37:14.685911 2201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa: Generated new TSK 0
I20260928 08:37:14.686841 2201 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:37:14.742971 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46501
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:15.106225 2215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:15.106547 2215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:15.106673 2215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:15.113818 2215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:15.113920 2215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:15.114006 2215 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 08:37:15.114068 2215 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:37:15.114130 2215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:37:15.124018 2215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:15.125254 2215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:15.126478 2215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:15.139604 2221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:15.139745 2220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:15.139715 2223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:15.140944 2215 server_base.cc:1034] running on GCE node
I20260928 08:37:15.141513 2215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:15.142537 2215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:15.143759 2215 hybrid_clock.cc:648] HybridClock initialized: now 1790584635143710 us; error 54 us; skew 500 ppm
I20260928 08:37:15.144073 2215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:15 dist-test-slave-w6c0 krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:15.147079 2215 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:37:15.148504 2215 webserver.cc:460] Webserver started at http://127.0.97.1:41371/ using document root <none> and password file <none>
I20260928 08:37:15.149276 2215 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:15.149417 2215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:15.149724 2215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:15.152702 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "a5e281e87b1345cdab79e45f2ac9a89d"
format_stamp: "Formatted at 2026-09-28 08:37:15 on dist-test-slave-w6c0"
server_key: "f0145ceb470eb887a3462b197d5ba5f9"
server_key_iv: "2ec88dd36a94079b76ca92b07c17a78f"
server_key_version: "encryptionkey@0"
I20260928 08:37:15.153677 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a5e281e87b1345cdab79e45f2ac9a89d"
format_stamp: "Formatted at 2026-09-28 08:37:15 on dist-test-slave-w6c0"
server_key: "f0145ceb470eb887a3462b197d5ba5f9"
server_key_iv: "2ec88dd36a94079b76ca92b07c17a78f"
server_key_version: "encryptionkey@0"
I20260928 08:37:15.158659 2215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 08:37:15.161854 2230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:15.163177 2215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:37:15.163352 2215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "a5e281e87b1345cdab79e45f2ac9a89d"
format_stamp: "Formatted at 2026-09-28 08:37:15 on dist-test-slave-w6c0"
server_key: "f0145ceb470eb887a3462b197d5ba5f9"
server_key_iv: "2ec88dd36a94079b76ca92b07c17a78f"
server_key_version: "encryptionkey@0"
I20260928 08:37:15.163578 2215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:15.178928 2215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:15.183284 2215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:15.183584 2215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:15.186468 2215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:15.186654 2215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 08:37:15.186780 2215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:15.186872 2215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:37:15 dist-test-slave-w6c0 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:15.211463 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:15.186717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60685 (local address 127.0.97.62:46501)
0928 08:37:15.187210 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:15.187216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:15.188454 (+ 1238us) server_negotiation.cc:407] Connection header received
0928 08:37:15.190462 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:15.190467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:15.190592 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:15.190717 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:15.192958 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:15.193891 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:15.196877 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:15.197626 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:15.197764 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:15.203650 (+ 5886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:15.203658 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:15.203667 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:15.203717 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:15.207867 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:15.209031 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:15.209037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:15.209039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:15.209160 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:15.209618 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:15.209623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:15.209627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:15.209908 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:15.210065 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:15.210860 (+ 795us) server_negotiation.cc:299] Negotiation successful
0928 08:37:15.211102 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":256,"threads_started":1}
I20260928 08:37:15.213078 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:15.187387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46501 (local address 127.0.97.1:60685)
0928 08:37:15.188283 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:15.188332 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:15.190185 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:15.190949 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:15.190958 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:15.191745 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:15.192695 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:15.192727 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:15.194051 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:15.194060 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:15.196678 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:15.196696 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:15.198030 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:15.198038 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:15.198407 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:15.199462 (+ 1055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:15.199497 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:15.203349 (+ 3852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:15.208060 (+ 4711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:15.208073 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:15.208118 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:15.208840 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:15.209274 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:15.209279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:15.209283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:15.209490 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:15.210039 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:15.210047 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:15.210421 (+ 374us) client_negotiation.cc:769] Sending connection context
0928 08:37:15.210765 (+ 344us) client_negotiation.cc:240] Negotiation successful
0928 08:37:15.211345 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":382,"threads_started":1}
I20260928 08:37:15.236652 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:15.220059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38319 (local address 127.0.97.62:46501)
0928 08:37:15.220281 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:15.220289 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:15.221357 (+ 1068us) server_negotiation.cc:407] Connection header received
0928 08:37:15.221512 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:15.221516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:15.221580 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:15.221656 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:15.223237 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:15.223915 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:15.226483 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:15.227136 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:15.227301 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:15.230050 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:15.230055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:15.230058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:15.230088 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:15.233595 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:15.234395 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:15.234403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:15.234409 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:15.234616 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:15.235128 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:15.235136 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:15.235141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:15.235502 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:15.235654 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:15.236082 (+ 428us) server_negotiation.cc:299] Negotiation successful
0928 08:37:15.236298 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 08:37:15.236749 2314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:15.220039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46501 (local address 127.0.97.1:38319)
0928 08:37:15.221142 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:15.221169 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:15.221366 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:15.221728 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:15.221735 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:15.222527 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:15.223053 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:15.223071 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:15.224100 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:15.224109 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:15.226310 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:15.226330 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:15.227406 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:15.227417 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:15.227709 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:15.228512 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:15.228540 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:15.229866 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:15.233742 (+ 3876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:15.233746 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:15.233749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:15.234258 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:15.234771 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:15.234781 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:15.234788 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:15.234971 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:15.235637 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:15.235647 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:15.235817 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 08:37:15.235967 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 08:37:15.236277 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"thread_start_us":673,"threads_started":1}
I20260928 08:37:15.243048 2215 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36265
I20260928 08:37:15.243083 2350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36265 every 8 connection(s)
I20260928 08:37:15.244248 2215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:37:15.245235 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 2215
I20260928 08:37:15.245430 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:37:15.246243 388 external_mini_cluster.cc:1428] Setting key da3e76c16d2492ad896c013357718fd3
I20260928 08:37:15.246603 2351 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46501
I20260928 08:37:15.246845 2351 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:15.247573 2351 heartbeater.cc:511] Master 127.0.97.62:46501 requested a full tablet report, sending...
I20260928 08:37:15.249742 2144 ts_manager.cc:194] Registered new tserver with Master: a5e281e87b1345cdab79e45f2ac9a89d (127.0.97.1:36265)
I20260928 08:37:15.251919 2144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60685
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:37:15.289762 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46501
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:15.614459 2355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:15.614806 2355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:15.614954 2355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:15.624769 2355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:15.624873 2355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:15.624957 2355 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 08:37:15.625008 2355 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:37:15.625078 2355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:37:15.634495 2355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:15.635712 2355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:15.636961 2355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:15.649375 2363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:15.649489 2360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:15.649403 2361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:15.650136 2355 server_base.cc:1034] running on GCE node
I20260928 08:37:15.650835 2355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:15.651754 2355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:15.652967 2355 hybrid_clock.cc:648] HybridClock initialized: now 1790584635652892 us; error 61 us; skew 500 ppm
I20260928 08:37:15.653278 2355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:15 dist-test-slave-w6c0 krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:15.659713 2355 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:37:15.661090 2355 webserver.cc:460] Webserver started at http://127.0.97.2:43121/ using document root <none> and password file <none>
I20260928 08:37:15.661875 2355 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:15.662001 2355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:15.662299 2355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:15.665088 2355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "0f871a89a97a475bbca04ba62b8d3798"
format_stamp: "Formatted at 2026-09-28 08:37:15 on dist-test-slave-w6c0"
server_key: "56f3e5423378b7e0319852d062690ede"
server_key_iv: "5f465fc4b56268975830c877e4bc767c"
server_key_version: "encryptionkey@0"
I20260928 08:37:15.665853 2355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0f871a89a97a475bbca04ba62b8d3798"
format_stamp: "Formatted at 2026-09-28 08:37:15 on dist-test-slave-w6c0"
server_key: "56f3e5423378b7e0319852d062690ede"
server_key_iv: "5f465fc4b56268975830c877e4bc767c"
server_key_version: "encryptionkey@0"
I20260928 08:37:15.670873 2355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 08:37:15.674047 2370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:15.675279 2355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 08:37:15.675462 2355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "0f871a89a97a475bbca04ba62b8d3798"
format_stamp: "Formatted at 2026-09-28 08:37:15 on dist-test-slave-w6c0"
server_key: "56f3e5423378b7e0319852d062690ede"
server_key_iv: "5f465fc4b56268975830c877e4bc767c"
server_key_version: "encryptionkey@0"
I20260928 08:37:15.675657 2355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:15.702190 2355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:15.705910 2355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:15.706200 2355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:15.708750 2355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:15.708869 2355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:15.708981 2355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:15.709074 2355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:37:15 dist-test-slave-w6c0 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:15.731315 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:15.709431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53273 (local address 127.0.97.62:46501)
0928 08:37:15.709646 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:15.709654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:15.711085 (+ 1431us) server_negotiation.cc:407] Connection header received
0928 08:37:15.713047 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:15.713053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:15.713150 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:15.713277 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:15.715407 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:15.716758 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:15.719466 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:15.719882 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:15.720020 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:15.724934 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:15.724939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:15.724942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:15.724975 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:15.727921 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:15.728879 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:15.728886 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:15.728892 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:15.729063 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:15.729562 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:15.729568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:15.729571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:15.729876 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:15.730037 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:15.730736 (+ 699us) server_negotiation.cc:299] Negotiation successful
0928 08:37:15.730951 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:37:15.732322 2379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:15.710069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46501 (local address 127.0.97.2:53273)
0928 08:37:15.710918 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:15.710961 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:15.712781 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:15.713556 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:15.713568 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:15.714251 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:15.715095 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:15.715116 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:15.716958 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:15.716967 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:15.719281 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:15.719298 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:15.720145 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:15.720159 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:15.720374 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:15.721267 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:15.721300 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:15.724698 (+ 3398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:15.728063 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:15.728074 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:15.728092 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:15.728716 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:15.729201 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:15.729211 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:15.729216 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:15.729428 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:15.730015 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:15.730023 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:15.730351 (+ 328us) client_negotiation.cc:769] Sending connection context
0928 08:37:15.730640 (+ 289us) client_negotiation.cc:240] Negotiation successful
0928 08:37:15.731192 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":419,"threads_started":1}
I20260928 08:37:15.755422 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:15.741277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34659 (local address 127.0.97.62:46501)
0928 08:37:15.741458 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:15.741463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:15.741661 (+ 198us) server_negotiation.cc:407] Connection header received
0928 08:37:15.741934 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:15.741939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:15.742009 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:15.742091 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:15.743676 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:15.744708 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:15.746029 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:15.746433 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:15.746617 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:15.749052 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:15.749056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:15.749059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:15.749094 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:15.752497 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:15.753335 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:15.753343 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:15.753349 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:15.753511 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:15.753868 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:15.753875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:15.753879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:15.754200 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:15.754352 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:15.754833 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 08:37:15.755038 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 08:37:15.755554 2445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:15.740797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46501 (local address 127.0.97.2:34659)
0928 08:37:15.741556 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:15.741577 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:15.741771 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:15.742284 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:15.742288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:15.743040 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:15.743522 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:15.743534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:15.744843 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:15.744848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:15.745893 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:15.745903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:15.746658 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:15.746665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:15.746832 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:15.747692 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:15.747709 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:15.748832 (+ 1123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:15.752627 (+ 3795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:15.752632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:15.752636 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:15.753215 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:15.753626 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:15.753630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:15.753634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:15.753763 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:15.754317 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:15.754322 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:15.754555 (+ 233us) client_negotiation.cc:769] Sending connection context
0928 08:37:15.754743 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 08:37:15.755064 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":537,"threads_started":1}
I20260928 08:37:15.763418 2355 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41961
I20260928 08:37:15.763454 2489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41961 every 8 connection(s)
I20260928 08:37:15.764587 2355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:37:15.766489 2490 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46501
I20260928 08:37:15.766736 2490 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:15.767375 2490 heartbeater.cc:511] Master 127.0.97.62:46501 requested a full tablet report, sending...
I20260928 08:37:15.767895 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 2355
I20260928 08:37:15.768082 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:37:15.768627 388 external_mini_cluster.cc:1428] Setting key 7cd9cf6819529dca1bb278fa484324f4
I20260928 08:37:15.768836 2144 ts_manager.cc:194] Registered new tserver with Master: 0f871a89a97a475bbca04ba62b8d3798 (127.0.97.2:41961)
I20260928 08:37:15.769706 2144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53273
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:37:15.809687 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46501
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:16.124799 2494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:16.125065 2494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:16.125226 2494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:16.132215 2494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:16.132311 2494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:16.132471 2494 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 08:37:16.132551 2494 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:37:16.132680 2494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:37:16.142385 2494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:16.143599 2494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:16.145097 2494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:16.157995 2499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:16.158016 2500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:16.158318 2502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:16.158847 2494 server_base.cc:1034] running on GCE node
I20260928 08:37:16.159394 2494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:16.160257 2494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:16.161477 2494 hybrid_clock.cc:648] HybridClock initialized: now 1790584636161437 us; error 50 us; skew 500 ppm
I20260928 08:37:16.161773 2494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:16 dist-test-slave-w6c0 krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:16.164774 2494 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:37:16.166088 2494 webserver.cc:460] Webserver started at http://127.0.97.3:43171/ using document root <none> and password file <none>
I20260928 08:37:16.167030 2494 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:16.167174 2494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:16.167478 2494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:16.170327 2494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "8c9be510e4c9479a8c073cb54caa2d73"
format_stamp: "Formatted at 2026-09-28 08:37:16 on dist-test-slave-w6c0"
server_key: "315133be453e65b8454f61f5a28c1012"
server_key_iv: "7202b9cd7967decf8114687b9843244f"
server_key_version: "encryptionkey@0"
I20260928 08:37:16.171198 2494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8c9be510e4c9479a8c073cb54caa2d73"
format_stamp: "Formatted at 2026-09-28 08:37:16 on dist-test-slave-w6c0"
server_key: "315133be453e65b8454f61f5a28c1012"
server_key_iv: "7202b9cd7967decf8114687b9843244f"
server_key_version: "encryptionkey@0"
I20260928 08:37:16.176373 2494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:37:16.179908 2509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:16.181190 2494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:37:16.181365 2494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "8c9be510e4c9479a8c073cb54caa2d73"
format_stamp: "Formatted at 2026-09-28 08:37:16 on dist-test-slave-w6c0"
server_key: "315133be453e65b8454f61f5a28c1012"
server_key_iv: "7202b9cd7967decf8114687b9843244f"
server_key_version: "encryptionkey@0"
I20260928 08:37:16.181568 2494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:16.196818 2494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:16.201433 2494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:16.201738 2494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:16.204316 2494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:16.204466 2494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:16.204594 2494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:16.204692 2494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:37:16 dist-test-slave-w6c0 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:16.228477 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.205652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46501 (local address 127.0.97.3:60087)
0928 08:37:16.206563 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:16.206614 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:16.208725 (+ 2111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:16.209528 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:16.209539 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:16.210043 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:16.210675 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.210693 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.212067 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.212079 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.214350 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.214366 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.215373 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.215381 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.215737 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.216553 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:16.216576 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:16.220221 (+ 3645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:16.223551 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:16.223564 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:16.223586 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:16.224237 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:16.224751 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:16.224754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:16.224756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:16.224965 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:16.225526 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:16.225534 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:16.226030 (+ 496us) client_negotiation.cc:769] Sending connection context
0928 08:37:16.226346 (+ 316us) client_negotiation.cc:240] Negotiation successful
0928 08:37:16.226901 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":396,"threads_started":1}
I20260928 08:37:16.229591 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.205117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60087 (local address 127.0.97.62:46501)
0928 08:37:16.205310 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:16.205317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:16.206739 (+ 1422us) server_negotiation.cc:407] Connection header received
0928 08:37:16.209013 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:16.209022 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:16.209135 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:16.209264 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:16.210850 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.211905 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.214530 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.215109 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.215261 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.220471 (+ 5210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:16.220475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:16.220478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:16.220510 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:16.223396 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:16.224404 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:16.224413 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:16.224420 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:16.224612 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:16.225096 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:16.225102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:16.225105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:16.225385 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:16.225563 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:16.229195 (+ 3632us) server_negotiation.cc:299] Negotiation successful
0928 08:37:16.229358 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 08:37:16.251601 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.236350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43569 (local address 127.0.97.62:46501)
0928 08:37:16.236581 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:16.236590 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:16.237131 (+ 541us) server_negotiation.cc:407] Connection header received
0928 08:37:16.237257 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:16.237265 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:16.237403 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:16.237520 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:16.238388 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.239644 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.242245 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.242789 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.242918 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.244960 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:16.244965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:16.244968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:16.244995 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:16.248247 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:16.249056 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:16.249059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:16.249062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:16.249171 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:16.249651 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:16.249658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:16.249665 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:16.250018 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:16.250173 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:16.250872 (+ 699us) server_negotiation.cc:299] Negotiation successful
0928 08:37:16.251124 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 08:37:16.253196 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.236395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46501 (local address 127.0.97.3:43569)
0928 08:37:16.237021 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:16.237044 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:16.237159 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:16.237538 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:16.237541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:16.237990 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:16.238252 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.238260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.239835 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.239844 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.242078 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.242095 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.243011 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.243018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.243175 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.243804 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:16.243828 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:16.244773 (+ 945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:16.248414 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:16.248420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:16.248424 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:16.248928 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:16.249309 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:16.249316 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:16.249322 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:16.249495 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:16.250189 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:16.250200 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:16.250462 (+ 262us) client_negotiation.cc:769] Sending connection context
0928 08:37:16.251881 (+ 1419us) client_negotiation.cc:240] Negotiation successful
0928 08:37:16.252385 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":330,"threads_started":1}
I20260928 08:37:16.255204 2351 heartbeater.cc:503] Master 127.0.97.62:46501 was elected leader, sending a full tablet report...
I20260928 08:37:16.258193 2494 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34833
I20260928 08:37:16.258214 2628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34833 every 8 connection(s)
I20260928 08:37:16.259289 2494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:37:16.261127 2629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46501
I20260928 08:37:16.261343 2629 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:16.261943 2629 heartbeater.cc:511] Master 127.0.97.62:46501 requested a full tablet report, sending...
I20260928 08:37:16.263165 2143 ts_manager.cc:194] Registered new tserver with Master: 8c9be510e4c9479a8c073cb54caa2d73 (127.0.97.3:34833)
I20260928 08:37:16.264020 2143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60087
I20260928 08:37:16.266685 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 2494
I20260928 08:37:16.266862 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:37:16.267336 388 external_mini_cluster.cc:1428] Setting key 1b7b19946f144f926f654bdf88a63a38
I20260928 08:37:16.269999 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:37:16 dist-test-slave-w6c0 krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:37:16 dist-test-slave-w6c0 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:16.314905 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.302062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58256 (local address 127.0.97.62:46501)
0928 08:37:16.302287 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:16.302294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:16.302469 (+ 175us) server_negotiation.cc:407] Connection header received
0928 08:37:16.302690 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:16.302694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:16.302758 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:16.302842 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:16.304080 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.304778 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.305887 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.306288 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.306374 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.310097 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:16.310101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:16.310104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:16.310136 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:16.312655 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:16.313247 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:16.313251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:16.313253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:16.313361 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:16.313701 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:16.313704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:16.313706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:16.313932 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:16.314101 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:16.314371 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 08:37:16.314596 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 08:37:16.319048 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:37:16.325802 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:37:16.351683 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.338038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36265 (local address 127.0.0.1:60856)
0928 08:37:16.339733 (+ 1695us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:16.339751 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:16.339827 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:16.341613 (+ 1786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:16.341621 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:16.341676 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:16.342275 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.342291 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.344866 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.344874 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.347932 (+ 3058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.347951 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.350479 (+ 2528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.350489 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.350770 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.350825 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 08:37:16.350904 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 08:37:16.351091 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1562,"mutex_wait_us":30,"spinlock_wait_cycles":896,"thread_start_us":1451,"threads_started":1}
I20260928 08:37:16.351931 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.338084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.0.97.2:41961)
0928 08:37:16.340652 (+ 2568us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:16.340660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:16.340690 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:37:16.340791 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:16.340800 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:16.341062 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:16.341256 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:16.342359 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.344390 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.347900 (+ 3510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.349468 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.349624 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.349913 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:16.351231 (+ 1318us) server_negotiation.cc:299] Negotiation successful
0928 08:37:16.351466 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2417,"thread_start_us":2281,"threads_started":1}
I20260928 08:37:16.353065 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.337967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41961 (local address 127.0.0.1:38178)
0928 08:37:16.339170 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:16.339215 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:16.339363 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:16.341494 (+ 2131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:16.341501 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:16.341533 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:16.342084 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.342103 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.344555 (+ 2452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.344563 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.347656 (+ 3093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.347676 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.350084 (+ 2408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.350094 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.350391 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.350465 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 08:37:16.350635 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 08:37:16.350912 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"mutex_wait_us":1,"spinlock_wait_cycles":1920,"thread_start_us":834,"threads_started":1}
I20260928 08:37:16.353358 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.338658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60856 (local address 127.0.97.1:36265)
0928 08:37:16.340989 (+ 2331us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:16.340998 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:16.341025 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:37:16.341118 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:16.341127 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:16.341333 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:16.342539 (+ 1206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:16.342784 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.344677 (+ 1893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.348178 (+ 3501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.350176 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.352448 (+ 2272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.352657 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:16.352726 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 08:37:16.352949 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2181,"thread_start_us":2053,"threads_started":1}
I20260928 08:37:16.356757 2420 tablet_service.cc:1467] Processing CreateTablet for tablet dc113a212d7645d08a2902d744599165 (DEFAULT_TABLE table=test-table [id=70658ee7fb954346b1681b23f1bc9376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:16.356817 2281 tablet_service.cc:1467] Processing CreateTablet for tablet dc113a212d7645d08a2902d744599165 (DEFAULT_TABLE table=test-table [id=70658ee7fb954346b1681b23f1bc9376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:16.358062 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc113a212d7645d08a2902d744599165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:16.358090 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc113a212d7645d08a2902d744599165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:16.362131 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.340576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34833 (local address 127.0.0.1:51048)
0928 08:37:16.352176 (+ 11600us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:16.352194 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:16.352287 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:16.355683 (+ 3396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:16.355689 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:16.355714 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:16.355864 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.355873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.357749 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.357757 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.359604 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.359618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.361376 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.361382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.361567 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.361613 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 08:37:16.361707 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 08:37:16.361841 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11511,"thread_start_us":12061,"threads_started":1}
I20260928 08:37:16.362234 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.341145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.0.97.3:34833)
0928 08:37:16.355059 (+ 13914us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:16.355070 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:16.355107 (+ 37us) server_negotiation.cc:407] Connection header received
0928 08:37:16.355192 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:16.355200 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:16.355471 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:16.355677 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:16.356079 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.357605 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.359796 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.361201 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.361328 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.361546 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:16.361772 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 08:37:16.361922 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13748,"thread_start_us":14769,"threads_started":1}
I20260928 08:37:16.365938 2559 tablet_service.cc:1467] Processing CreateTablet for tablet dc113a212d7645d08a2902d744599165 (DEFAULT_TABLE table=test-table [id=70658ee7fb954346b1681b23f1bc9376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:16.367439 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc113a212d7645d08a2902d744599165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:16.370211 2648 tablet_bootstrap.cc:492] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d: Bootstrap starting.
I20260928 08:37:16.372143 2649 tablet_bootstrap.cc:492] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798: Bootstrap starting.
I20260928 08:37:16.373920 2648 tablet_bootstrap.cc:654] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:16.375211 2648 log.cc:824] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:16.375447 2649 tablet_bootstrap.cc:654] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:16.376577 2649 log.cc:824] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:16.378170 2648 tablet_bootstrap.cc:492] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d: No bootstrap required, opened a new log
I20260928 08:37:16.378453 2648 ts_tablet_manager.cc:1397] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260928 08:37:16.379357 2649 tablet_bootstrap.cc:492] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798: No bootstrap required, opened a new log
I20260928 08:37:16.379608 2649 ts_tablet_manager.cc:1397] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260928 08:37:16.380721 2652 tablet_bootstrap.cc:492] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73: Bootstrap starting.
I20260928 08:37:16.383526 2652 tablet_bootstrap.cc:654] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:16.384570 2652 log.cc:824] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:16.387507 2652 tablet_bootstrap.cc:492] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73: No bootstrap required, opened a new log
I20260928 08:37:16.387758 2652 ts_tablet_manager.cc:1397] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260928 08:37:16.389624 2648 raft_consensus.cc:348] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:16.390048 2648 raft_consensus.cc:374] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:16.390192 2648 raft_consensus.cc:729] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e281e87b1345cdab79e45f2ac9a89d, State: Initialized, Role: FOLLOWER
I20260928 08:37:16.390167 2649 raft_consensus.cc:348] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:16.390655 2649 raft_consensus.cc:374] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:16.390756 2649 raft_consensus.cc:729] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f871a89a97a475bbca04ba62b8d3798, State: Initialized, Role: FOLLOWER
I20260928 08:37:16.390833 2648 consensus_queue.cc:260] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:16.391302 2649 consensus_queue.cc:260] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:16.392602 2648 ts_tablet_manager.cc:1428] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260928 08:37:16.395166 2490 heartbeater.cc:503] Master 127.0.97.62:46501 was elected leader, sending a full tablet report...
I20260928 08:37:16.395787 2649 ts_tablet_manager.cc:1428] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260928 08:37:16.395851 2652 raft_consensus.cc:348] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:16.396139 2652 raft_consensus.cc:374] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:16.396263 2652 raft_consensus.cc:729] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9be510e4c9479a8c073cb54caa2d73, State: Initialized, Role: FOLLOWER
I20260928 08:37:16.396800 2652 consensus_queue.cc:260] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:16.397816 2629 heartbeater.cc:503] Master 127.0.97.62:46501 was elected leader, sending a full tablet report...
I20260928 08:37:16.398435 2652 ts_tablet_manager.cc:1428] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
W20260928 08:37:16.496860 2352 tablet.cc:2369] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:37:16.510842 2630 tablet.cc:2369] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:37:16.516562 2491 tablet.cc:2369] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:37:16.521761 2656 raft_consensus.cc:482] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:37:16.522054 2656 raft_consensus.cc:504] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:16.523902 2656 leader_election.cc:288] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f871a89a97a475bbca04ba62b8d3798 (127.0.97.2:41961), a5e281e87b1345cdab79e45f2ac9a89d (127.0.97.1:36265)
I20260928 08:37:16.530406 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.524358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48465 (local address 127.0.97.2:41961)
0928 08:37:16.524562 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:16.524567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:16.524588 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:37:16.524660 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:16.524663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:16.524718 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:16.524820 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:16.525358 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.526446 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.528404 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.529608 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.529704 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.529784 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:16.530120 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 08:37:16.530219 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:37:16.530524 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.524057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41961 (local address 127.0.97.3:48465)
0928 08:37:16.524324 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:16.524344 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:16.524479 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:16.524837 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:16.524841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:16.524874 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:16.525151 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.525170 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.526612 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.526617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.528232 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.528247 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.529758 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.529764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.529947 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.529981 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 08:37:16.530078 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 08:37:16.530216 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260928 08:37:16.531632 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.524750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50173 (local address 127.0.97.1:36265)
0928 08:37:16.525011 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:16.525021 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:16.525552 (+ 531us) server_negotiation.cc:407] Connection header received
0928 08:37:16.525664 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:16.525668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:16.525746 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:16.525834 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:16.526386 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.527482 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.529522 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.530745 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.530884 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.531016 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:16.531271 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 08:37:16.531395 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:37:16.531711 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.524687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36265 (local address 127.0.97.3:50173)
0928 08:37:16.525404 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:16.525422 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:16.525542 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:16.525850 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:16.525854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:16.525880 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:16.526228 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.526241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.527609 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.527612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.529361 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.529376 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.530942 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.530948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.531123 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.531145 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 08:37:16.531245 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 08:37:16.531360 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":396,"threads_started":1}
I20260928 08:37:16.531450 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc113a212d7645d08a2902d744599165" candidate_uuid: "8c9be510e4c9479a8c073cb54caa2d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f871a89a97a475bbca04ba62b8d3798" is_pre_election: true
I20260928 08:37:16.532083 2444 raft_consensus.cc:2454] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c9be510e4c9479a8c073cb54caa2d73 in term 0.
I20260928 08:37:16.532787 2513 leader_election.cc:302] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f871a89a97a475bbca04ba62b8d3798, 8c9be510e4c9479a8c073cb54caa2d73; no voters:
I20260928 08:37:16.532647 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc113a212d7645d08a2902d744599165" candidate_uuid: "8c9be510e4c9479a8c073cb54caa2d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" is_pre_election: true
I20260928 08:37:16.533191 2656 raft_consensus.cc:2781] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:37:16.533192 2304 raft_consensus.cc:2454] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c9be510e4c9479a8c073cb54caa2d73 in term 0.
I20260928 08:37:16.533344 2656 raft_consensus.cc:482] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:37:16.533468 2656 raft_consensus.cc:3037] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:16.535890 2656 raft_consensus.cc:504] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:16.536487 2656 leader_election.cc:288] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [CANDIDATE]: Term 1 election: Requested vote from peers 0f871a89a97a475bbca04ba62b8d3798 (127.0.97.2:41961), a5e281e87b1345cdab79e45f2ac9a89d (127.0.97.1:36265)
I20260928 08:37:16.536931 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc113a212d7645d08a2902d744599165" candidate_uuid: "8c9be510e4c9479a8c073cb54caa2d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f871a89a97a475bbca04ba62b8d3798"
I20260928 08:37:16.537177 2444 raft_consensus.cc:3037] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:16.537191 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc113a212d7645d08a2902d744599165" candidate_uuid: "8c9be510e4c9479a8c073cb54caa2d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e281e87b1345cdab79e45f2ac9a89d"
I20260928 08:37:16.537431 2304 raft_consensus.cc:3037] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:16.539638 2444 raft_consensus.cc:2454] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c9be510e4c9479a8c073cb54caa2d73 in term 1.
I20260928 08:37:16.539685 2304 raft_consensus.cc:2454] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c9be510e4c9479a8c073cb54caa2d73 in term 1.
I20260928 08:37:16.540017 2513 leader_election.cc:302] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f871a89a97a475bbca04ba62b8d3798, 8c9be510e4c9479a8c073cb54caa2d73; no voters:
I20260928 08:37:16.540299 2656 raft_consensus.cc:2781] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:16.540742 2656 raft_consensus.cc:686] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 LEADER]: Becoming Leader. State: Replica: 8c9be510e4c9479a8c073cb54caa2d73, State: Running, Role: LEADER
I20260928 08:37:16.541205 2656 consensus_queue.cc:237] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:16.545804 2142 catalog_manager.cc:5495] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c9be510e4c9479a8c073cb54caa2d73 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8c9be510e4c9479a8c073cb54caa2d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } health_report { overall_health: HEALTHY } } }
Sep 28 08:37:16 dist-test-slave-w6c0 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:16.612628 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.596980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.0.97.62:46501)
0928 08:37:16.597216 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:16.597225 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:16.597833 (+ 608us) server_negotiation.cc:407] Connection header received
0928 08:37:16.598034 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:16.598038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:16.598099 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:16.598183 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:16.599848 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.600703 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.602622 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.603065 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.603176 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.607217 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:16.607222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:16.607225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:16.607254 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:16.609893 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:16.610722 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:16.610726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:16.610729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:16.610851 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:16.611244 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:16.611247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:16.611249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:16.611492 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:16.611614 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:16.612082 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 08:37:16.612375 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":96}
I20260928 08:37:16.612713 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.597015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46501 (local address 127.0.0.1:58272)
0928 08:37:16.597705 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:16.597726 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:16.597878 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:16.598254 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:16.598264 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:16.599284 (+ 1020us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:16.599647 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.599658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.600840 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.600846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.602436 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.602449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.603268 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.603275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.603451 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.604183 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:16.604205 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:16.607031 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:16.610024 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:16.610037 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:16.610046 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:16.610602 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:16.610957 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:16.610962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:16.610966 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:16.611135 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:16.611619 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:16.611627 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:16.611816 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 08:37:16.611981 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 08:37:16.612260 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":328,"threads_started":1}
I20260928 08:37:16.618448 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58272:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 08:37:16.619606 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:37:16.626008 2281 tablet_service.cc:1467] Processing CreateTablet for tablet c86f76b2c04f4c0387a7709f69db34e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2a89271bda3b44a6bbd650ee1683a530]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:37:16.626186 2559 tablet_service.cc:1467] Processing CreateTablet for tablet c86f76b2c04f4c0387a7709f69db34e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2a89271bda3b44a6bbd650ee1683a530]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:37:16.626186 2420 tablet_service.cc:1467] Processing CreateTablet for tablet c86f76b2c04f4c0387a7709f69db34e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2a89271bda3b44a6bbd650ee1683a530]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:37:16.627055 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86f76b2c04f4c0387a7709f69db34e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:16.627056 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86f76b2c04f4c0387a7709f69db34e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:16.627056 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86f76b2c04f4c0387a7709f69db34e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:16.633014 2648 tablet_bootstrap.cc:492] T c86f76b2c04f4c0387a7709f69db34e6 P a5e281e87b1345cdab79e45f2ac9a89d: Bootstrap starting.
I20260928 08:37:16.633984 2652 tablet_bootstrap.cc:492] T c86f76b2c04f4c0387a7709f69db34e6 P 8c9be510e4c9479a8c073cb54caa2d73: Bootstrap starting.
I20260928 08:37:16.634013 2649 tablet_bootstrap.cc:492] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798: Bootstrap starting.
I20260928 08:37:16.634684 2648 tablet_bootstrap.cc:654] T c86f76b2c04f4c0387a7709f69db34e6 P a5e281e87b1345cdab79e45f2ac9a89d: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:16.635659 2652 tablet_bootstrap.cc:654] T c86f76b2c04f4c0387a7709f69db34e6 P 8c9be510e4c9479a8c073cb54caa2d73: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:16.635854 2649 tablet_bootstrap.cc:654] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:16.637555 2648 tablet_bootstrap.cc:492] T c86f76b2c04f4c0387a7709f69db34e6 P a5e281e87b1345cdab79e45f2ac9a89d: No bootstrap required, opened a new log
I20260928 08:37:16.637691 2648 ts_tablet_manager.cc:1397] T c86f76b2c04f4c0387a7709f69db34e6 P a5e281e87b1345cdab79e45f2ac9a89d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260928 08:37:16.638211 2652 tablet_bootstrap.cc:492] T c86f76b2c04f4c0387a7709f69db34e6 P 8c9be510e4c9479a8c073cb54caa2d73: No bootstrap required, opened a new log
I20260928 08:37:16.638367 2652 ts_tablet_manager.cc:1397] T c86f76b2c04f4c0387a7709f69db34e6 P 8c9be510e4c9479a8c073cb54caa2d73: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260928 08:37:16.638491 2648 raft_consensus.cc:348] T c86f76b2c04f4c0387a7709f69db34e6 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:16.638756 2649 tablet_bootstrap.cc:492] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798: No bootstrap required, opened a new log
I20260928 08:37:16.638772 2648 raft_consensus.cc:374] T c86f76b2c04f4c0387a7709f69db34e6 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:16.638862 2648 raft_consensus.cc:729] T c86f76b2c04f4c0387a7709f69db34e6 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e281e87b1345cdab79e45f2ac9a89d, State: Initialized, Role: FOLLOWER
I20260928 08:37:16.638942 2649 ts_tablet_manager.cc:1397] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 08:37:16.639094 2648 consensus_queue.cc:260] T c86f76b2c04f4c0387a7709f69db34e6 P a5e281e87b1345cdab79e45f2ac9a89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:16.639670 2648 ts_tablet_manager.cc:1428] T c86f76b2c04f4c0387a7709f69db34e6 P a5e281e87b1345cdab79e45f2ac9a89d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 08:37:16.639648 2649 raft_consensus.cc:348] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:16.639565 2654 tablet_replica.cc:439] T c86f76b2c04f4c0387a7709f69db34e6 P a5e281e87b1345cdab79e45f2ac9a89d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } } }
I20260928 08:37:16.639876 2649 raft_consensus.cc:374] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:16.639922 2654 tablet_replica.cc:442] T c86f76b2c04f4c0387a7709f69db34e6 P a5e281e87b1345cdab79e45f2ac9a89d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:37:16.639971 2649 raft_consensus.cc:729] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f871a89a97a475bbca04ba62b8d3798, State: Initialized, Role: FOLLOWER
I20260928 08:37:16.640232 2649 consensus_queue.cc:260] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:16.640813 2649 ts_tablet_manager.cc:1428] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 08:37:16.640651 2655 tablet_replica.cc:439] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } } }
I20260928 08:37:16.641048 2655 tablet_replica.cc:442] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:37:16.639223 2652 raft_consensus.cc:348] T c86f76b2c04f4c0387a7709f69db34e6 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:16.641866 2652 raft_consensus.cc:374] T c86f76b2c04f4c0387a7709f69db34e6 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:16.642017 2652 raft_consensus.cc:729] T c86f76b2c04f4c0387a7709f69db34e6 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9be510e4c9479a8c073cb54caa2d73, State: Initialized, Role: FOLLOWER
I20260928 08:37:16.642277 2652 consensus_queue.cc:260] T c86f76b2c04f4c0387a7709f69db34e6 P 8c9be510e4c9479a8c073cb54caa2d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:16.642881 2652 ts_tablet_manager.cc:1428] T c86f76b2c04f4c0387a7709f69db34e6 P 8c9be510e4c9479a8c073cb54caa2d73: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 08:37:16.642828 2656 tablet_replica.cc:439] T c86f76b2c04f4c0387a7709f69db34e6 P 8c9be510e4c9479a8c073cb54caa2d73: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } } }
I20260928 08:37:16.643092 2656 tablet_replica.cc:442] T c86f76b2c04f4c0387a7709f69db34e6 P 8c9be510e4c9479a8c073cb54caa2d73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:37:16.822177 2655 raft_consensus.cc:482] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:37:16.822397 2655 raft_consensus.cc:504] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:16.823992 2655 leader_election.cc:288] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5e281e87b1345cdab79e45f2ac9a89d (127.0.97.1:36265), 8c9be510e4c9479a8c073cb54caa2d73 (127.0.97.3:34833)
I20260928 08:37:16.830868 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.824389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43163 (local address 127.0.97.3:34833)
0928 08:37:16.824590 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:16.824599 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:16.825467 (+ 868us) server_negotiation.cc:407] Connection header received
0928 08:37:16.825665 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:16.825669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:16.825727 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:16.825822 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:16.826437 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.827179 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.829077 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.829820 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.829980 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.830092 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:16.830424 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 08:37:16.830598 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:37:16.830883 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.824376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37133 (local address 127.0.97.1:36265)
0928 08:37:16.824629 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:16.824638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:16.825277 (+ 639us) server_negotiation.cc:407] Connection header received
0928 08:37:16.825496 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:16.825502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:16.825581 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:16.825679 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:16.826290 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.827131 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.829063 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.829846 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.830000 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.830109 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:16.830444 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 08:37:16.830600 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 08:37:16.831063 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.824292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36265 (local address 127.0.97.2:37133)
0928 08:37:16.825174 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:16.825189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:16.825356 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:16.825705 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:16.825709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:16.825738 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:16.826125 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.826135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.827282 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.827287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.828908 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.828919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.830076 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.830081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.830271 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.830298 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 08:37:16.830397 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 08:37:16.830586 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":143,"thread_start_us":654,"threads_started":1}
I20260928 08:37:16.831475 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.824381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34833 (local address 127.0.97.2:43163)
0928 08:37:16.825353 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:16.825367 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:16.825524 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:16.825838 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:16.825842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:16.825878 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:16.826267 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.826276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.827302 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.827310 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.828923 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.828934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.830042 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.830048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.830260 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.830291 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 08:37:16.830396 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 08:37:16.830584 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"mutex_wait_us":88,"thread_start_us":396,"threads_started":1}
I20260928 08:37:16.831815 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86f76b2c04f4c0387a7709f69db34e6" candidate_uuid: "0f871a89a97a475bbca04ba62b8d3798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" is_pre_election: true
I20260928 08:37:16.832087 2304 raft_consensus.cc:2454] T c86f76b2c04f4c0387a7709f69db34e6 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f871a89a97a475bbca04ba62b8d3798 in term 0.
I20260928 08:37:16.832451 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86f76b2c04f4c0387a7709f69db34e6" candidate_uuid: "0f871a89a97a475bbca04ba62b8d3798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c9be510e4c9479a8c073cb54caa2d73" is_pre_election: true
I20260928 08:37:16.832650 2373 leader_election.cc:302] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f871a89a97a475bbca04ba62b8d3798, a5e281e87b1345cdab79e45f2ac9a89d; no voters:
I20260928 08:37:16.832808 2583 raft_consensus.cc:2454] T c86f76b2c04f4c0387a7709f69db34e6 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f871a89a97a475bbca04ba62b8d3798 in term 0.
I20260928 08:37:16.833046 2655 raft_consensus.cc:2781] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:37:16.833181 2655 raft_consensus.cc:482] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:37:16.833313 2655 raft_consensus.cc:3037] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:16.835666 2655 raft_consensus.cc:504] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:16.836217 2655 leader_election.cc:288] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 [CANDIDATE]: Term 1 election: Requested vote from peers a5e281e87b1345cdab79e45f2ac9a89d (127.0.97.1:36265), 8c9be510e4c9479a8c073cb54caa2d73 (127.0.97.3:34833)
I20260928 08:37:16.836695 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86f76b2c04f4c0387a7709f69db34e6" candidate_uuid: "0f871a89a97a475bbca04ba62b8d3798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e281e87b1345cdab79e45f2ac9a89d"
I20260928 08:37:16.836683 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86f76b2c04f4c0387a7709f69db34e6" candidate_uuid: "0f871a89a97a475bbca04ba62b8d3798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c9be510e4c9479a8c073cb54caa2d73"
I20260928 08:37:16.836884 2583 raft_consensus.cc:3037] T c86f76b2c04f4c0387a7709f69db34e6 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:16.836907 2304 raft_consensus.cc:3037] T c86f76b2c04f4c0387a7709f69db34e6 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:16.839838 2304 raft_consensus.cc:2454] T c86f76b2c04f4c0387a7709f69db34e6 P a5e281e87b1345cdab79e45f2ac9a89d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f871a89a97a475bbca04ba62b8d3798 in term 1.
I20260928 08:37:16.839848 2583 raft_consensus.cc:2454] T c86f76b2c04f4c0387a7709f69db34e6 P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f871a89a97a475bbca04ba62b8d3798 in term 1.
I20260928 08:37:16.840225 2373 leader_election.cc:302] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f871a89a97a475bbca04ba62b8d3798, a5e281e87b1345cdab79e45f2ac9a89d; no voters:
I20260928 08:37:16.840530 2655 raft_consensus.cc:2781] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:16.841001 2655 raft_consensus.cc:686] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 [term 1 LEADER]: Becoming Leader. State: Replica: 0f871a89a97a475bbca04ba62b8d3798, State: Running, Role: LEADER
I20260928 08:37:16.841512 2655 consensus_queue.cc:237] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:16.844683 2671 tablet_replica.cc:439] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798: TxnStatusTablet state changed. Reason: New leader 0f871a89a97a475bbca04ba62b8d3798. Latest consensus state: current_term: 1 leader_uuid: "0f871a89a97a475bbca04ba62b8d3798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } } }
I20260928 08:37:16.845113 2671 tablet_replica.cc:442] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:37:16.845799 2673 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 08:37:16.846087 2673 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 08:37:16.846058 2142 catalog_manager.cc:5495] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f871a89a97a475bbca04ba62b8d3798 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0f871a89a97a475bbca04ba62b8d3798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } health_report { overall_health: UNKNOWN } } }
I20260928 08:37:16.874752 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.869236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41961 (local address 127.0.0.1:38194)
0928 08:37:16.869458 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:16.869474 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:16.869621 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:16.870103 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:16.870108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:16.870120 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:37:16.870296 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.870304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.871427 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.871435 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.872404 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.872414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.873127 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.873133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.873298 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.873349 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:37:16.874193 (+ 844us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:37:16.874212 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 08:37:16.874283 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 08:37:16.874431 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 08:37:16.874787 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.869455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38194 (local address 127.0.97.2:41961)
0928 08:37:16.869640 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:16.869651 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:16.869672 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:37:16.869848 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:16.869853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:16.869937 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:16.870085 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:37:16.870422 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.871292 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.872549 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.872967 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.873101 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.873555 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:37:16.874077 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:37:16.874251 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:16.874313 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 08:37:16.874393 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:37:16.882930 2583 raft_consensus.cc:1261] T c86f76b2c04f4c0387a7709f69db34e6 P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 FOLLOWER]: Refusing update from remote peer 0f871a89a97a475bbca04ba62b8d3798: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:16.883822 2304 raft_consensus.cc:1261] T c86f76b2c04f4c0387a7709f69db34e6 P a5e281e87b1345cdab79e45f2ac9a89d [term 1 FOLLOWER]: Refusing update from remote peer 0f871a89a97a475bbca04ba62b8d3798: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:16.884755 2671 consensus_queue.cc:1035] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:16.889246 2671 consensus_queue.cc:1035] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:16.890950 2656 tablet_replica.cc:439] T c86f76b2c04f4c0387a7709f69db34e6 P 8c9be510e4c9479a8c073cb54caa2d73: TxnStatusTablet state changed. Reason: New leader 0f871a89a97a475bbca04ba62b8d3798. Latest consensus state: current_term: 1 leader_uuid: "0f871a89a97a475bbca04ba62b8d3798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } } }
I20260928 08:37:16.891243 2656 tablet_replica.cc:442] T c86f76b2c04f4c0387a7709f69db34e6 P 8c9be510e4c9479a8c073cb54caa2d73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:37:16.895123 2676 mvcc.cc:205] Tried to move back new op lower bound from 7334234672661803008 to 7334234672506961920. Current Snapshot: MvccSnapshot[applied={T|T < 7334234672661803008}]
I20260928 08:37:16.896588 2656 tablet_replica.cc:439] T c86f76b2c04f4c0387a7709f69db34e6 P 8c9be510e4c9479a8c073cb54caa2d73: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f871a89a97a475bbca04ba62b8d3798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } } }
I20260928 08:37:16.896921 2656 tablet_replica.cc:442] T c86f76b2c04f4c0387a7709f69db34e6 P 8c9be510e4c9479a8c073cb54caa2d73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:37:16.898365 2677 mvcc.cc:205] Tried to move back new op lower bound from 7334234672661803008 to 7334234672506961920. Current Snapshot: MvccSnapshot[applied={T|T < 7334234672661803008}]
I20260928 08:37:16.901356 2655 tablet_replica.cc:439] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f871a89a97a475bbca04ba62b8d3798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } } }
I20260928 08:37:16.901700 2655 tablet_replica.cc:442] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:37:16.903864 2654 tablet_replica.cc:439] T c86f76b2c04f4c0387a7709f69db34e6 P a5e281e87b1345cdab79e45f2ac9a89d: TxnStatusTablet state changed. Reason: New leader 0f871a89a97a475bbca04ba62b8d3798. Latest consensus state: current_term: 1 leader_uuid: "0f871a89a97a475bbca04ba62b8d3798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } } }
I20260928 08:37:16.904178 2654 tablet_replica.cc:442] T c86f76b2c04f4c0387a7709f69db34e6 P a5e281e87b1345cdab79e45f2ac9a89d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:37:16.907318 2671 tablet_replica.cc:439] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f871a89a97a475bbca04ba62b8d3798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } } }
I20260928 08:37:16.907672 2671 tablet_replica.cc:442] T c86f76b2c04f4c0387a7709f69db34e6 P 0f871a89a97a475bbca04ba62b8d3798: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:37:16.912330 2654 tablet_replica.cc:439] T c86f76b2c04f4c0387a7709f69db34e6 P a5e281e87b1345cdab79e45f2ac9a89d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f871a89a97a475bbca04ba62b8d3798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } } }
I20260928 08:37:16.912560 2654 tablet_replica.cc:442] T c86f76b2c04f4c0387a7709f69db34e6 P a5e281e87b1345cdab79e45f2ac9a89d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:37:16.924938 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.912808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51050 (local address 127.0.97.3:34833)
0928 08:37:16.912997 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:16.913004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:16.919078 (+ 6074us) server_negotiation.cc:407] Connection header received
0928 08:37:16.919241 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:16.919248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:16.919369 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:16.919531 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:37:16.919901 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.921063 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.922418 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.923090 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.923218 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.923579 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:37:16.924313 (+ 734us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:37:16.924457 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:16.924533 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 08:37:16.924617 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:37:16.941584 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.936068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59503 (local address 127.0.97.2:41961)
0928 08:37:16.936257 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:16.936262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:16.936634 (+ 372us) server_negotiation.cc:407] Connection header received
0928 08:37:16.936841 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:16.936847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:16.936938 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:16.937037 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:37:16.937559 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.938457 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.939822 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.940270 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.940409 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.940839 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:37:16.940952 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:37:16.941054 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:16.941183 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 08:37:16.941260 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:37:16.941666 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.935943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41961 (local address 127.0.97.3:59503)
0928 08:37:16.936532 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:16.936552 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:16.936707 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:16.937049 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:16.937053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:16.937064 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:37:16.937390 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.937399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.938644 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.938650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.939681 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.939691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.940439 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.940445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.940583 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.940633 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:37:16.941052 (+ 419us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:37:16.941064 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 08:37:16.941162 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 08:37:16.941291 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":289,"threads_started":1}
I20260928 08:37:16.952153 2304 raft_consensus.cc:1261] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d [term 1 FOLLOWER]: Refusing update from remote peer 8c9be510e4c9479a8c073cb54caa2d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:16.952373 2444 raft_consensus.cc:1261] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798 [term 1 FOLLOWER]: Refusing update from remote peer 8c9be510e4c9479a8c073cb54caa2d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:16.953210 2658 consensus_queue.cc:1035] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:16.953954 2656 consensus_queue.cc:1035] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:16.988281 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.980818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42025 (local address 127.0.97.3:34833)
0928 08:37:16.980989 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:16.980993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:16.981718 (+ 725us) server_negotiation.cc:407] Connection header received
0928 08:37:16.981918 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:16.981923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:16.982008 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:16.982095 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:37:16.982897 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.984027 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.986270 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.986978 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.987078 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.987699 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:37:16.987800 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:37:16.987870 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:16.988023 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 08:37:16.988083 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 08:37:16.988811 2693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:16.980807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34833 (local address 127.0.97.2:42025)
0928 08:37:16.981605 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:16.981625 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:16.981769 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:16.982257 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:16.982262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:16.982276 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:37:16.982715 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.982727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.984198 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.984203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.986103 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:16.986123 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:16.987273 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:16.987279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:16.987442 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:16.987523 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:37:16.987899 (+ 376us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:37:16.987916 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 08:37:16.988013 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 08:37:16.988149 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":414,"threads_started":1}
I20260928 08:37:17.035866 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:17.036167 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:17.040475 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa: Sending DeleteTablet for 3 replicas of tablet dc113a212d7645d08a2902d744599165
I20260928 08:37:17.041641 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet dc113a212d7645d08a2902d744599165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38178
I20260928 08:37:17.042671 2697 tablet_replica.cc:330] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798: stopping tablet replica
I20260928 08:37:17.042822 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet dc113a212d7645d08a2902d744599165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60856
I20260928 08:37:17.043104 2697 raft_consensus.cc:2229] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:17.043453 2697 raft_consensus.cc:2258] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:17.043865 2559 tablet_service.cc:1514] Processing DeleteTablet for tablet dc113a212d7645d08a2902d744599165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51048
I20260928 08:37:17.045492 2697 ts_tablet_manager.cc:1905] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:17.052350 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:37:17.053042 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:37:17.054292 2697 ts_tablet_manager.cc:1918] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:37:17.054474 2697 log.cc:1196] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/dc113a212d7645d08a2902d744599165
I20260928 08:37:17.055100 2697 ts_tablet_manager.cc:1939] T dc113a212d7645d08a2902d744599165 P 0f871a89a97a475bbca04ba62b8d3798: Deleting consensus metadata
I20260928 08:37:17.061703 2420 tablet_service.cc:1467] Processing CreateTablet for tablet 8bad0e913ded46e6aa44bedbe4c4168b (DEFAULT_TABLE table=test-table [id=59a833056450415096a901d9a648b15a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:17.061785 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 8bad0e913ded46e6aa44bedbe4c4168b (DEFAULT_TABLE table=test-table [id=59a833056450415096a901d9a648b15a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:17.062198 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bad0e913ded46e6aa44bedbe4c4168b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:17.062273 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bad0e913ded46e6aa44bedbe4c4168b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:17.063803 2131 catalog_manager.cc:4841] TS 0f871a89a97a475bbca04ba62b8d3798 (127.0.97.2:41961): tablet dc113a212d7645d08a2902d744599165 (table test-table [id=70658ee7fb954346b1681b23f1bc9376]) successfully deleted
I20260928 08:37:17.067335 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 8bad0e913ded46e6aa44bedbe4c4168b (DEFAULT_TABLE table=test-table [id=59a833056450415096a901d9a648b15a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:17.067870 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bad0e913ded46e6aa44bedbe4c4168b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:17.068965 2652 tablet_bootstrap.cc:492] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73: Bootstrap starting.
I20260928 08:37:17.070744 2652 tablet_bootstrap.cc:654] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:17.083303 2652 tablet_bootstrap.cc:492] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73: No bootstrap required, opened a new log
I20260928 08:37:17.083544 2652 ts_tablet_manager.cc:1397] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20260928 08:37:17.084597 2652 raft_consensus.cc:348] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:17.084926 2652 raft_consensus.cc:374] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:17.085019 2652 raft_consensus.cc:729] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9be510e4c9479a8c073cb54caa2d73, State: Initialized, Role: FOLLOWER
I20260928 08:37:17.085312 2652 consensus_queue.cc:260] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:17.087218 2652 ts_tablet_manager.cc:1428] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 08:37:17.093925 2648 tablet_bootstrap.cc:492] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d: Bootstrap starting.
I20260928 08:37:17.095551 2649 tablet_bootstrap.cc:492] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798: Bootstrap starting.
I20260928 08:37:17.096130 2648 tablet_bootstrap.cc:654] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:17.097368 2649 tablet_bootstrap.cc:654] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:17.108500 2649 tablet_bootstrap.cc:492] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798: No bootstrap required, opened a new log
I20260928 08:37:17.108769 2649 ts_tablet_manager.cc:1397] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260928 08:37:17.109227 2699 tablet_replica.cc:330] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73: stopping tablet replica
I20260928 08:37:17.109746 2698 tablet_replica.cc:330] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d: stopping tablet replica
I20260928 08:37:17.110291 2699 raft_consensus.cc:2229] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:37:17.110797 2698 raft_consensus.cc:2229] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:17.111073 2649 raft_consensus.cc:348] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:17.111474 2649 raft_consensus.cc:374] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:17.111600 2649 raft_consensus.cc:729] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f871a89a97a475bbca04ba62b8d3798, State: Initialized, Role: FOLLOWER
I20260928 08:37:17.111896 2699 raft_consensus.cc:2258] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:17.112118 2698 raft_consensus.cc:2258] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:17.111966 2649 consensus_queue.cc:260] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:17.113260 2649 ts_tablet_manager.cc:1428] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260928 08:37:17.114897 2648 tablet_bootstrap.cc:492] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d: No bootstrap required, opened a new log
I20260928 08:37:17.115140 2648 ts_tablet_manager.cc:1397] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.004s
I20260928 08:37:17.115970 2648 raft_consensus.cc:348] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:17.116433 2648 raft_consensus.cc:374] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:17.116578 2648 raft_consensus.cc:729] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e281e87b1345cdab79e45f2ac9a89d, State: Initialized, Role: FOLLOWER
I20260928 08:37:17.116887 2648 consensus_queue.cc:260] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:17.117628 2648 ts_tablet_manager.cc:1428] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:37:17.119036 2699 ts_tablet_manager.cc:1905] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:17.126732 2699 ts_tablet_manager.cc:1918] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:37:17.126914 2699 log.cc:1196] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/dc113a212d7645d08a2902d744599165
I20260928 08:37:17.127458 2699 ts_tablet_manager.cc:1939] T dc113a212d7645d08a2902d744599165 P 8c9be510e4c9479a8c073cb54caa2d73: Deleting consensus metadata
I20260928 08:37:17.128499 2698 ts_tablet_manager.cc:1905] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:17.129099 2128 catalog_manager.cc:4841] TS 8c9be510e4c9479a8c073cb54caa2d73 (127.0.97.3:34833): tablet dc113a212d7645d08a2902d744599165 (table test-table [id=70658ee7fb954346b1681b23f1bc9376]) successfully deleted
I20260928 08:37:17.137269 2698 ts_tablet_manager.cc:1918] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:37:17.137540 2698 log.cc:1196] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/wal/wals/dc113a212d7645d08a2902d744599165
I20260928 08:37:17.138136 2698 ts_tablet_manager.cc:1939] T dc113a212d7645d08a2902d744599165 P a5e281e87b1345cdab79e45f2ac9a89d: Deleting consensus metadata
I20260928 08:37:17.140072 2130 catalog_manager.cc:4841] TS a5e281e87b1345cdab79e45f2ac9a89d (127.0.97.1:36265): tablet dc113a212d7645d08a2902d744599165 (table test-table [id=70658ee7fb954346b1681b23f1bc9376]) successfully deleted
I20260928 08:37:17.174039 2656 raft_consensus.cc:482] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:37:17.174261 2656 raft_consensus.cc:504] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:17.175094 2656 leader_election.cc:288] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5e281e87b1345cdab79e45f2ac9a89d (127.0.97.1:36265), 0f871a89a97a475bbca04ba62b8d3798 (127.0.97.2:41961)
I20260928 08:37:17.175681 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bad0e913ded46e6aa44bedbe4c4168b" candidate_uuid: "8c9be510e4c9479a8c073cb54caa2d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" is_pre_election: true
I20260928 08:37:17.175825 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bad0e913ded46e6aa44bedbe4c4168b" candidate_uuid: "8c9be510e4c9479a8c073cb54caa2d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f871a89a97a475bbca04ba62b8d3798" is_pre_election: true
I20260928 08:37:17.175932 2304 raft_consensus.cc:2454] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c9be510e4c9479a8c073cb54caa2d73 in term 0.
I20260928 08:37:17.176080 2444 raft_consensus.cc:2454] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c9be510e4c9479a8c073cb54caa2d73 in term 0.
I20260928 08:37:17.176601 2513 leader_election.cc:302] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f871a89a97a475bbca04ba62b8d3798, 8c9be510e4c9479a8c073cb54caa2d73, a5e281e87b1345cdab79e45f2ac9a89d; no voters:
I20260928 08:37:17.176942 2656 raft_consensus.cc:2781] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:37:17.177083 2656 raft_consensus.cc:482] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:37:17.177186 2656 raft_consensus.cc:3037] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:17.180449 2656 raft_consensus.cc:504] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:17.181676 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bad0e913ded46e6aa44bedbe4c4168b" candidate_uuid: "8c9be510e4c9479a8c073cb54caa2d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f871a89a97a475bbca04ba62b8d3798"
I20260928 08:37:17.181680 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bad0e913ded46e6aa44bedbe4c4168b" candidate_uuid: "8c9be510e4c9479a8c073cb54caa2d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e281e87b1345cdab79e45f2ac9a89d"
I20260928 08:37:17.181885 2304 raft_consensus.cc:3037] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:17.181905 2444 raft_consensus.cc:3037] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:17.185215 2444 raft_consensus.cc:2454] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c9be510e4c9479a8c073cb54caa2d73 in term 1.
I20260928 08:37:17.185230 2304 raft_consensus.cc:2454] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c9be510e4c9479a8c073cb54caa2d73 in term 1.
I20260928 08:37:17.185779 2512 leader_election.cc:302] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c9be510e4c9479a8c073cb54caa2d73, a5e281e87b1345cdab79e45f2ac9a89d; no voters:
I20260928 08:37:17.186087 2658 raft_consensus.cc:2781] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:17.186249 2658 raft_consensus.cc:686] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 LEADER]: Becoming Leader. State: Replica: 8c9be510e4c9479a8c073cb54caa2d73, State: Running, Role: LEADER
I20260928 08:37:17.186309 2656 leader_election.cc:288] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [CANDIDATE]: Term 1 election: Requested vote from peers a5e281e87b1345cdab79e45f2ac9a89d (127.0.97.1:36265), 0f871a89a97a475bbca04ba62b8d3798 (127.0.97.2:41961)
I20260928 08:37:17.186574 2658 consensus_queue.cc:237] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:17.190855 2144 catalog_manager.cc:5495] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c9be510e4c9479a8c073cb54caa2d73 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8c9be510e4c9479a8c073cb54caa2d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } health_report { overall_health: HEALTHY } } }
I20260928 08:37:17.241240 2304 raft_consensus.cc:1261] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d [term 1 FOLLOWER]: Refusing update from remote peer 8c9be510e4c9479a8c073cb54caa2d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:17.241896 2658 consensus_queue.cc:1035] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:17.252398 2444 raft_consensus.cc:1261] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798 [term 1 FOLLOWER]: Refusing update from remote peer 8c9be510e4c9479a8c073cb54caa2d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:17.253252 2658 consensus_queue.cc:1035] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:17.273741 2679 mvcc.cc:205] Tried to move back new op lower bound from 7334234674134622208 to 7334234673919201280. Current Snapshot: MvccSnapshot[applied={T|T < 7334234674134622208}]
W20260928 08:37:17.308157 2677 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 08:37:18.914906 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:18.915104 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:18.918881 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa: Sending DeleteTablet for 3 replicas of tablet 8bad0e913ded46e6aa44bedbe4c4168b
I20260928 08:37:18.919584 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bad0e913ded46e6aa44bedbe4c4168b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38178
I20260928 08:37:18.919584 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bad0e913ded46e6aa44bedbe4c4168b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60856
I20260928 08:37:18.919706 2559 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bad0e913ded46e6aa44bedbe4c4168b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51048
I20260928 08:37:18.932483 2732 tablet_replica.cc:330] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798: stopping tablet replica
I20260928 08:37:18.932761 2731 tablet_replica.cc:330] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d: stopping tablet replica
I20260928 08:37:18.932907 2732 raft_consensus.cc:2229] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:18.933034 2731 raft_consensus.cc:2229] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:18.933267 2732 raft_consensus.cc:2258] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:18.933367 2731 raft_consensus.cc:2258] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:18.933946 2733 tablet_replica.cc:330] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73: stopping tablet replica
I20260928 08:37:18.934237 2733 raft_consensus.cc:2229] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:37:18.934459 2732 ts_tablet_manager.cc:1905] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:18.934459 2731 ts_tablet_manager.cc:1905] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:18.934747 2733 raft_consensus.cc:2258] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:18.935812 2733 ts_tablet_manager.cc:1905] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:18.941262 2731 ts_tablet_manager.cc:1918] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:37:18.941417 2731 log.cc:1196] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/wal/wals/8bad0e913ded46e6aa44bedbe4c4168b
I20260928 08:37:18.941651 2732 ts_tablet_manager.cc:1918] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:37:18.941789 2732 log.cc:1196] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/8bad0e913ded46e6aa44bedbe4c4168b
I20260928 08:37:18.941870 2731 ts_tablet_manager.cc:1939] T 8bad0e913ded46e6aa44bedbe4c4168b P a5e281e87b1345cdab79e45f2ac9a89d: Deleting consensus metadata
I20260928 08:37:18.942226 2732 ts_tablet_manager.cc:1939] T 8bad0e913ded46e6aa44bedbe4c4168b P 0f871a89a97a475bbca04ba62b8d3798: Deleting consensus metadata
I20260928 08:37:18.943034 2130 catalog_manager.cc:4841] TS a5e281e87b1345cdab79e45f2ac9a89d (127.0.97.1:36265): tablet 8bad0e913ded46e6aa44bedbe4c4168b (table test-table [id=59a833056450415096a901d9a648b15a]) successfully deleted
I20260928 08:37:18.943261 2131 catalog_manager.cc:4841] TS 0f871a89a97a475bbca04ba62b8d3798 (127.0.97.2:41961): tablet 8bad0e913ded46e6aa44bedbe4c4168b (table test-table [id=59a833056450415096a901d9a648b15a]) successfully deleted
I20260928 08:37:18.943573 2733 ts_tablet_manager.cc:1918] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:37:18.943713 2733 log.cc:1196] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/8bad0e913ded46e6aa44bedbe4c4168b
I20260928 08:37:18.944170 2733 ts_tablet_manager.cc:1939] T 8bad0e913ded46e6aa44bedbe4c4168b P 8c9be510e4c9479a8c073cb54caa2d73: Deleting consensus metadata
I20260928 08:37:18.945258 2128 catalog_manager.cc:4841] TS 8c9be510e4c9479a8c073cb54caa2d73 (127.0.97.3:34833): tablet 8bad0e913ded46e6aa44bedbe4c4168b (table test-table [id=59a833056450415096a901d9a648b15a]) successfully deleted
Sep 28 08:37:18 dist-test-slave-w6c0 krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584638, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:37:18 dist-test-slave-w6c0 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584638, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:18.977756 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:18.963332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58274 (local address 127.0.97.62:46501)
0928 08:37:18.963890 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:18.963900 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:18.963972 (+ 72us) server_negotiation.cc:407] Connection header received
0928 08:37:18.964196 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:18.964200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:18.964261 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:18.964380 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:18.966030 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:18.966856 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:18.968302 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:18.968680 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:18.968773 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:18.972558 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:18.972562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:18.972565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:18.972597 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:18.975403 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:18.975973 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:18.975977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:18.975979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:18.976097 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:18.976389 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:18.976392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:18.976394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:18.976709 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:18.976903 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:18.977219 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 08:37:18.977376 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":252,"threads_started":1}
I20260928 08:37:18.981312 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:37:18.981949 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:37:18.988238 2281 tablet_service.cc:1467] Processing CreateTablet for tablet 6704c04e893d4bda8a0800d1e51db214 (DEFAULT_TABLE table=test-table [id=676c1079c24745f2911c7773497bcd64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:18.988353 2420 tablet_service.cc:1467] Processing CreateTablet for tablet 6704c04e893d4bda8a0800d1e51db214 (DEFAULT_TABLE table=test-table [id=676c1079c24745f2911c7773497bcd64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:18.988448 2559 tablet_service.cc:1467] Processing CreateTablet for tablet 6704c04e893d4bda8a0800d1e51db214 (DEFAULT_TABLE table=test-table [id=676c1079c24745f2911c7773497bcd64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:18.988691 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6704c04e893d4bda8a0800d1e51db214. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:18.988737 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6704c04e893d4bda8a0800d1e51db214. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:18.988818 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6704c04e893d4bda8a0800d1e51db214. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:18.994482 2742 tablet_bootstrap.cc:492] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798: Bootstrap starting.
I20260928 08:37:18.995313 2743 tablet_bootstrap.cc:492] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d: Bootstrap starting.
I20260928 08:37:18.995603 2744 tablet_bootstrap.cc:492] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73: Bootstrap starting.
I20260928 08:37:18.996183 2742 tablet_bootstrap.cc:654] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:18.996941 2743 tablet_bootstrap.cc:654] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:18.997118 2744 tablet_bootstrap.cc:654] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:18.999266 2742 tablet_bootstrap.cc:492] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798: No bootstrap required, opened a new log
I20260928 08:37:18.999402 2742 ts_tablet_manager.cc:1397] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260928 08:37:18.999581 2744 tablet_bootstrap.cc:492] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73: No bootstrap required, opened a new log
I20260928 08:37:18.999717 2744 ts_tablet_manager.cc:1397] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 08:37:19.000119 2742 raft_consensus.cc:348] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:19.000291 2742 raft_consensus.cc:374] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:19.000388 2742 raft_consensus.cc:729] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f871a89a97a475bbca04ba62b8d3798, State: Initialized, Role: FOLLOWER
I20260928 08:37:19.000315 2744 raft_consensus.cc:348] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:19.000557 2744 raft_consensus.cc:374] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:19.000643 2744 raft_consensus.cc:729] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9be510e4c9479a8c073cb54caa2d73, State: Initialized, Role: FOLLOWER
I20260928 08:37:19.000633 2742 consensus_queue.cc:260] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:19.000870 2744 consensus_queue.cc:260] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:19.001130 2742 ts_tablet_manager.cc:1428] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 08:37:19.001400 2744 ts_tablet_manager.cc:1428] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 08:37:19.002132 2743 tablet_bootstrap.cc:492] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d: No bootstrap required, opened a new log
I20260928 08:37:19.002285 2743 ts_tablet_manager.cc:1397] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260928 08:37:19.003074 2743 raft_consensus.cc:348] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:19.003314 2743 raft_consensus.cc:374] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:19.003412 2743 raft_consensus.cc:729] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e281e87b1345cdab79e45f2ac9a89d, State: Initialized, Role: FOLLOWER
I20260928 08:37:19.003666 2743 consensus_queue.cc:260] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:19.004438 2743 ts_tablet_manager.cc:1428] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 08:37:19.056213 2722 raft_consensus.cc:482] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:37:19.056420 2722 raft_consensus.cc:504] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:19.057129 2722 leader_election.cc:288] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5e281e87b1345cdab79e45f2ac9a89d (127.0.97.1:36265), 0f871a89a97a475bbca04ba62b8d3798 (127.0.97.2:41961)
I20260928 08:37:19.057638 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6704c04e893d4bda8a0800d1e51db214" candidate_uuid: "8c9be510e4c9479a8c073cb54caa2d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" is_pre_election: true
I20260928 08:37:19.057701 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6704c04e893d4bda8a0800d1e51db214" candidate_uuid: "8c9be510e4c9479a8c073cb54caa2d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f871a89a97a475bbca04ba62b8d3798" is_pre_election: true
I20260928 08:37:19.057915 2302 raft_consensus.cc:2454] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c9be510e4c9479a8c073cb54caa2d73 in term 0.
I20260928 08:37:19.057973 2442 raft_consensus.cc:2454] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c9be510e4c9479a8c073cb54caa2d73 in term 0.
I20260928 08:37:19.058401 2512 leader_election.cc:302] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c9be510e4c9479a8c073cb54caa2d73, a5e281e87b1345cdab79e45f2ac9a89d; no voters:
I20260928 08:37:19.058681 2722 raft_consensus.cc:2781] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:37:19.058804 2722 raft_consensus.cc:482] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:37:19.058874 2722 raft_consensus.cc:3037] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:19.061328 2722 raft_consensus.cc:504] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:19.061902 2722 leader_election.cc:288] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [CANDIDATE]: Term 1 election: Requested vote from peers a5e281e87b1345cdab79e45f2ac9a89d (127.0.97.1:36265), 0f871a89a97a475bbca04ba62b8d3798 (127.0.97.2:41961)
I20260928 08:37:19.062379 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6704c04e893d4bda8a0800d1e51db214" candidate_uuid: "8c9be510e4c9479a8c073cb54caa2d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e281e87b1345cdab79e45f2ac9a89d"
I20260928 08:37:19.062435 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6704c04e893d4bda8a0800d1e51db214" candidate_uuid: "8c9be510e4c9479a8c073cb54caa2d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f871a89a97a475bbca04ba62b8d3798"
I20260928 08:37:19.062613 2302 raft_consensus.cc:3037] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:19.062666 2442 raft_consensus.cc:3037] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:19.065503 2302 raft_consensus.cc:2454] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c9be510e4c9479a8c073cb54caa2d73 in term 1.
I20260928 08:37:19.065503 2442 raft_consensus.cc:2454] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c9be510e4c9479a8c073cb54caa2d73 in term 1.
I20260928 08:37:19.065876 2512 leader_election.cc:302] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c9be510e4c9479a8c073cb54caa2d73, a5e281e87b1345cdab79e45f2ac9a89d; no voters:
I20260928 08:37:19.066242 2722 raft_consensus.cc:2781] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:19.066406 2722 raft_consensus.cc:686] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 LEADER]: Becoming Leader. State: Replica: 8c9be510e4c9479a8c073cb54caa2d73, State: Running, Role: LEADER
I20260928 08:37:19.066929 2722 consensus_queue.cc:237] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:19.069686 2143 catalog_manager.cc:5495] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c9be510e4c9479a8c073cb54caa2d73 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8c9be510e4c9479a8c073cb54caa2d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } health_report { overall_health: HEALTHY } } }
I20260928 08:37:19.105321 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:19.099733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51058 (local address 127.0.97.3:34833)
0928 08:37:19.100479 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:19.100483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:19.100500 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:37:19.100544 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:19.100548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:19.100602 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:19.100677 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:37:19.101028 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.101743 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.103188 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.103599 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.103745 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:19.104178 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:37:19.104280 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:37:19.104380 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:19.104533 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 08:37:19.104608 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":523,"threads_started":1}
I20260928 08:37:19.113796 2302 raft_consensus.cc:1261] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d [term 1 FOLLOWER]: Refusing update from remote peer 8c9be510e4c9479a8c073cb54caa2d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:19.113806 2442 raft_consensus.cc:1261] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798 [term 1 FOLLOWER]: Refusing update from remote peer 8c9be510e4c9479a8c073cb54caa2d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:19.114320 2722 consensus_queue.cc:1035] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:19.114645 2709 consensus_queue.cc:1035] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:19.156569 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:19.156767 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:19.159355 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa: Sending DeleteTablet for 3 replicas of tablet 6704c04e893d4bda8a0800d1e51db214
I20260928 08:37:19.160041 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet 6704c04e893d4bda8a0800d1e51db214 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60856
I20260928 08:37:19.160149 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet 6704c04e893d4bda8a0800d1e51db214 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38178
I20260928 08:37:19.160175 2559 tablet_service.cc:1514] Processing DeleteTablet for tablet 6704c04e893d4bda8a0800d1e51db214 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51048
I20260928 08:37:19.160313 2731 tablet_replica.cc:330] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d: stopping tablet replica
I20260928 08:37:19.160439 2732 tablet_replica.cc:330] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798: stopping tablet replica
I20260928 08:37:19.160538 2731 raft_consensus.cc:2229] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:19.160625 2732 raft_consensus.cc:2229] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:19.160719 2731 raft_consensus.cc:2258] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:19.160789 2732 raft_consensus.cc:2258] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:19.161204 2733 tablet_replica.cc:330] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73: stopping tablet replica
I20260928 08:37:19.161398 2733 raft_consensus.cc:2229] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:37:19.161687 2733 raft_consensus.cc:2258] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:19.161432 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 08:37:19.161813 2731 ts_tablet_manager.cc:1905] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:19.161813 2732 ts_tablet_manager.cc:1905] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 08:37:19.162153 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:37:19.162966 2733 ts_tablet_manager.cc:1905] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:19.168020 2733 ts_tablet_manager.cc:1918] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:37:19.168202 2733 log.cc:1196] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/6704c04e893d4bda8a0800d1e51db214
I20260928 08:37:19.168339 2731 ts_tablet_manager.cc:1918] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:37:19.168352 2732 ts_tablet_manager.cc:1918] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:37:19.168491 2732 log.cc:1196] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/6704c04e893d4bda8a0800d1e51db214
I20260928 08:37:19.168491 2731 log.cc:1196] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/wal/wals/6704c04e893d4bda8a0800d1e51db214
I20260928 08:37:19.168672 2733 ts_tablet_manager.cc:1939] T 6704c04e893d4bda8a0800d1e51db214 P 8c9be510e4c9479a8c073cb54caa2d73: Deleting consensus metadata
I20260928 08:37:19.168902 2731 ts_tablet_manager.cc:1939] T 6704c04e893d4bda8a0800d1e51db214 P a5e281e87b1345cdab79e45f2ac9a89d: Deleting consensus metadata
I20260928 08:37:19.168902 2732 ts_tablet_manager.cc:1939] T 6704c04e893d4bda8a0800d1e51db214 P 0f871a89a97a475bbca04ba62b8d3798: Deleting consensus metadata
I20260928 08:37:19.169632 2128 catalog_manager.cc:4841] TS 8c9be510e4c9479a8c073cb54caa2d73 (127.0.97.3:34833): tablet 6704c04e893d4bda8a0800d1e51db214 (table test-table [id=676c1079c24745f2911c7773497bcd64]) successfully deleted
I20260928 08:37:19.169823 2130 catalog_manager.cc:4841] TS a5e281e87b1345cdab79e45f2ac9a89d (127.0.97.1:36265): tablet 6704c04e893d4bda8a0800d1e51db214 (table test-table [id=676c1079c24745f2911c7773497bcd64]) successfully deleted
I20260928 08:37:19.170336 2131 catalog_manager.cc:4841] TS 0f871a89a97a475bbca04ba62b8d3798 (127.0.97.2:41961): tablet 6704c04e893d4bda8a0800d1e51db214 (table test-table [id=676c1079c24745f2911c7773497bcd64]) successfully deleted
I20260928 08:37:19.174196 2281 tablet_service.cc:1467] Processing CreateTablet for tablet be1196c6f2e9409a81c7d5c1db083212 (DEFAULT_TABLE table=test-table [id=b5b9253f598944d88f2902679447e230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:19.174367 2420 tablet_service.cc:1467] Processing CreateTablet for tablet be1196c6f2e9409a81c7d5c1db083212 (DEFAULT_TABLE table=test-table [id=b5b9253f598944d88f2902679447e230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:19.174536 2559 tablet_service.cc:1467] Processing CreateTablet for tablet be1196c6f2e9409a81c7d5c1db083212 (DEFAULT_TABLE table=test-table [id=b5b9253f598944d88f2902679447e230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:19.174628 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1196c6f2e9409a81c7d5c1db083212. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:19.174798 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1196c6f2e9409a81c7d5c1db083212. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:19.174901 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1196c6f2e9409a81c7d5c1db083212. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:19.179512 2742 tablet_bootstrap.cc:492] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798: Bootstrap starting.
I20260928 08:37:19.180802 2742 tablet_bootstrap.cc:654] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:19.180903 2744 tablet_bootstrap.cc:492] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73: Bootstrap starting.
I20260928 08:37:19.180918 2743 tablet_bootstrap.cc:492] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d: Bootstrap starting.
I20260928 08:37:19.182169 2743 tablet_bootstrap.cc:654] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:19.182863 2744 tablet_bootstrap.cc:654] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:19.183344 2742 tablet_bootstrap.cc:492] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798: No bootstrap required, opened a new log
I20260928 08:37:19.183501 2742 ts_tablet_manager.cc:1397] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 08:37:19.184286 2743 tablet_bootstrap.cc:492] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d: No bootstrap required, opened a new log
I20260928 08:37:19.184453 2743 ts_tablet_manager.cc:1397] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260928 08:37:19.184514 2742 raft_consensus.cc:348] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:19.184863 2742 raft_consensus.cc:374] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:19.184959 2742 raft_consensus.cc:729] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f871a89a97a475bbca04ba62b8d3798, State: Initialized, Role: FOLLOWER
I20260928 08:37:19.185158 2743 raft_consensus.cc:348] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:19.185427 2743 raft_consensus.cc:374] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:19.185264 2742 consensus_queue.cc:260] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:19.185506 2743 raft_consensus.cc:729] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e281e87b1345cdab79e45f2ac9a89d, State: Initialized, Role: FOLLOWER
I20260928 08:37:19.185949 2742 ts_tablet_manager.cc:1428] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 08:37:19.186347 2743 consensus_queue.cc:260] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:19.187481 2743 ts_tablet_manager.cc:1428] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 08:37:19.187958 2744 tablet_bootstrap.cc:492] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73: No bootstrap required, opened a new log
I20260928 08:37:19.188127 2744 ts_tablet_manager.cc:1397] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260928 08:37:19.188871 2744 raft_consensus.cc:348] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:19.189479 2744 raft_consensus.cc:374] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:19.189571 2744 raft_consensus.cc:729] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9be510e4c9479a8c073cb54caa2d73, State: Initialized, Role: FOLLOWER
I20260928 08:37:19.189791 2744 consensus_queue.cc:260] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:19.190326 2744 ts_tablet_manager.cc:1428] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 08:37:19.193013 2748 raft_consensus.cc:482] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:37:19.193212 2748 raft_consensus.cc:504] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:19.195178 2748 leader_election.cc:288] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f871a89a97a475bbca04ba62b8d3798 (127.0.97.2:41961), 8c9be510e4c9479a8c073cb54caa2d73 (127.0.97.3:34833)
I20260928 08:37:19.204814 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:19.195497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53775 (local address 127.0.97.3:34833)
0928 08:37:19.195775 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:19.195807 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:19.197007 (+ 1200us) server_negotiation.cc:407] Connection header received
0928 08:37:19.197227 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:19.197236 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:19.197392 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:19.197540 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:19.198102 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.200177 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.203116 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.203861 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.204012 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:19.204113 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:19.204453 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 08:37:19.204580 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 08:37:19.205104 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:19.196175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34833 (local address 127.0.97.1:53775)
0928 08:37:19.196910 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:19.196930 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:19.197058 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:19.197550 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:19.197554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:19.197578 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:19.197889 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:19.197900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.200408 (+ 2508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.200416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:19.202970 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:19.202981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.204047 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.204053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:19.204237 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:19.204271 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 08:37:19.204426 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 08:37:19.204555 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":363,"threads_started":1}
I20260928 08:37:19.205695 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1196c6f2e9409a81c7d5c1db083212" candidate_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c9be510e4c9479a8c073cb54caa2d73" is_pre_election: true
I20260928 08:37:19.205927 2583 raft_consensus.cc:2454] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e281e87b1345cdab79e45f2ac9a89d in term 0.
I20260928 08:37:19.206419 2231 leader_election.cc:302] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c9be510e4c9479a8c073cb54caa2d73, a5e281e87b1345cdab79e45f2ac9a89d; no voters:
I20260928 08:37:19.206835 2748 raft_consensus.cc:2781] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:37:19.206960 2748 raft_consensus.cc:482] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:37:19.207052 2748 raft_consensus.cc:3037] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:19.208567 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:19.195345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41961 (local address 127.0.97.1:43145)
0928 08:37:19.195868 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:19.195894 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:19.196033 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:19.203124 (+ 7091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:19.203128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:19.203154 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:19.203474 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:19.203482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.204710 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.204715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:19.206299 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:19.206311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.207994 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.207998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:19.208134 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:19.208160 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 08:37:19.208233 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 08:37:19.208331 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":405,"threads_started":1}
I20260928 08:37:19.208752 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:19.195650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43145 (local address 127.0.97.2:41961)
0928 08:37:19.202874 (+ 7224us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:19.202878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:19.202898 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:37:19.202944 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:19.202948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:19.202996 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:19.203112 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:19.203622 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.204558 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.206453 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.207763 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.207897 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:19.208000 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:19.208279 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 08:37:19.208421 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7127,"thread_start_us":7030,"threads_started":1}
I20260928 08:37:19.209617 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1196c6f2e9409a81c7d5c1db083212" candidate_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f871a89a97a475bbca04ba62b8d3798" is_pre_election: true
I20260928 08:37:19.209851 2442 raft_consensus.cc:2454] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e281e87b1345cdab79e45f2ac9a89d in term 0.
I20260928 08:37:19.210920 2748 raft_consensus.cc:504] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:19.211467 2748 leader_election.cc:288] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [CANDIDATE]: Term 1 election: Requested vote from peers 0f871a89a97a475bbca04ba62b8d3798 (127.0.97.2:41961), 8c9be510e4c9479a8c073cb54caa2d73 (127.0.97.3:34833)
I20260928 08:37:19.211921 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1196c6f2e9409a81c7d5c1db083212" candidate_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c9be510e4c9479a8c073cb54caa2d73"
I20260928 08:37:19.211916 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1196c6f2e9409a81c7d5c1db083212" candidate_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f871a89a97a475bbca04ba62b8d3798"
I20260928 08:37:19.212109 2442 raft_consensus.cc:3037] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:19.212109 2583 raft_consensus.cc:3037] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:19.214957 2583 raft_consensus.cc:2454] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e281e87b1345cdab79e45f2ac9a89d in term 1.
I20260928 08:37:19.214957 2442 raft_consensus.cc:2454] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e281e87b1345cdab79e45f2ac9a89d in term 1.
I20260928 08:37:19.215338 2231 leader_election.cc:302] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c9be510e4c9479a8c073cb54caa2d73, a5e281e87b1345cdab79e45f2ac9a89d; no voters:
I20260928 08:37:19.215620 2748 raft_consensus.cc:2781] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:19.216065 2748 raft_consensus.cc:686] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [term 1 LEADER]: Becoming Leader. State: Replica: a5e281e87b1345cdab79e45f2ac9a89d, State: Running, Role: LEADER
I20260928 08:37:19.216504 2748 consensus_queue.cc:237] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } }
I20260928 08:37:19.219782 2142 catalog_manager.cc:5495] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d reported cstate change: term changed from 0 to 1, leader changed from <none> to a5e281e87b1345cdab79e45f2ac9a89d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e281e87b1345cdab79e45f2ac9a89d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 } health_report { overall_health: UNKNOWN } } }
I20260928 08:37:19.243001 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:19.237026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60868 (local address 127.0.97.1:36265)
0928 08:37:19.237978 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:19.237982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:19.237998 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:37:19.238077 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:19.238080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:19.238146 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:19.238289 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:37:19.238814 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.239613 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.240843 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.241275 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.241407 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:19.241748 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:37:19.242305 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:37:19.242407 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:19.242584 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 08:37:19.242652 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":856,"thread_start_us":744,"threads_started":1}
I20260928 08:37:19.259581 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:19.253100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54343 (local address 127.0.97.2:41961)
0928 08:37:19.253285 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:19.253291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:19.253729 (+ 438us) server_negotiation.cc:407] Connection header received
0928 08:37:19.253934 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:19.253940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:19.254034 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:19.254127 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:37:19.254699 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.255744 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.257516 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.258180 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.258323 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:19.258901 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:37:19.259025 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:37:19.259106 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:19.259246 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 08:37:19.259324 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 08:37:19.259754 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:19.252982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41961 (local address 127.0.97.1:54343)
0928 08:37:19.253641 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:19.253657 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:19.253807 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:19.254197 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:19.254200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:19.254216 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:37:19.254536 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:19.254555 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.256290 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.256293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:19.257369 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:19.257379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.258389 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.258395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:19.258637 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:19.258718 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:37:19.259124 (+ 406us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:37:19.259134 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 08:37:19.259210 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 08:37:19.259348 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":315,"threads_started":1}
I20260928 08:37:19.268476 2442 raft_consensus.cc:1261] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798 [term 1 FOLLOWER]: Refusing update from remote peer a5e281e87b1345cdab79e45f2ac9a89d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:19.268760 2583 raft_consensus.cc:1261] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 FOLLOWER]: Refusing update from remote peer a5e281e87b1345cdab79e45f2ac9a89d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:19.269446 2762 consensus_queue.cc:1035] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f871a89a97a475bbca04ba62b8d3798" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:19.270073 2762 consensus_queue.cc:1035] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c9be510e4c9479a8c073cb54caa2d73" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:19.314919 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:19.307864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42621 (local address 127.0.97.1:36265)
0928 08:37:19.308016 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:19.308021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:19.310056 (+ 2035us) server_negotiation.cc:407] Connection header received
0928 08:37:19.310190 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:19.310195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:19.310294 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:19.310383 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:37:19.310832 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.311883 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.313512 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.313915 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.314010 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:19.314320 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:37:19.314408 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:37:19.314473 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:19.314652 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 08:37:19.314717 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 08:37:19.314970 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:19.307834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36265 (local address 127.0.97.2:42621)
0928 08:37:19.309943 (+ 2109us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:19.309963 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:19.310093 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:19.310406 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:19.310409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:19.310417 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:37:19.310690 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:19.310698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.312011 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.312016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:19.313387 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:19.313399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:19.314042 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:19.314047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:19.314160 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:19.314193 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:37:19.314488 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:37:19.314495 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 08:37:19.314614 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 08:37:19.314705 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1966,"thread_start_us":1630,"threads_started":1}
W20260928 08:37:19.316362 2714 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 08:37:20.913532 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:20.913836 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:20.918246 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f386719bd68486f9689a343758a3bfa: Sending DeleteTablet for 3 replicas of tablet be1196c6f2e9409a81c7d5c1db083212
I20260928 08:37:20.919049 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet be1196c6f2e9409a81c7d5c1db083212 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60856
I20260928 08:37:20.919190 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet be1196c6f2e9409a81c7d5c1db083212 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38178
I20260928 08:37:20.919615 2559 tablet_service.cc:1514] Processing DeleteTablet for tablet be1196c6f2e9409a81c7d5c1db083212 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51048
I20260928 08:37:20.920239 2799 tablet_replica.cc:330] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798: stopping tablet replica
I20260928 08:37:20.920420 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 2215
I20260928 08:37:20.920548 2799 raft_consensus.cc:2229] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:20.920826 2799 raft_consensus.cc:2258] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:20.926994 2798 tablet_replica.cc:330] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d: stopping tablet replica
I20260928 08:37:20.927284 2798 raft_consensus.cc:2229] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:37:20.927546 2799 ts_tablet_manager.cc:1905] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:20.927925 2798 raft_consensus.cc:2258] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:20.929924 2800 tablet_replica.cc:330] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73: stopping tablet replica
I20260928 08:37:20.930209 2800 raft_consensus.cc:2229] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:20.930461 2798 ts_tablet_manager.cc:1905] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:20.930548 2800 raft_consensus.cc:2258] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:20.931533 2800 ts_tablet_manager.cc:1905] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:20.934927 2799 ts_tablet_manager.cc:1918] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:37:20.935135 2799 log.cc:1196] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/be1196c6f2e9409a81c7d5c1db083212
I20260928 08:37:20.935714 2799 ts_tablet_manager.cc:1939] T be1196c6f2e9409a81c7d5c1db083212 P 0f871a89a97a475bbca04ba62b8d3798: Deleting consensus metadata
I20260928 08:37:20.937402 2131 catalog_manager.cc:4841] TS 0f871a89a97a475bbca04ba62b8d3798 (127.0.97.2:41961): tablet be1196c6f2e9409a81c7d5c1db083212 (table test-table [id=b5b9253f598944d88f2902679447e230]) successfully deleted
I20260928 08:37:20.937654 2798 ts_tablet_manager.cc:1918] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:37:20.937795 2798 log.cc:1196] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-0/wal/wals/be1196c6f2e9409a81c7d5c1db083212
I20260928 08:37:20.938100 2800 ts_tablet_manager.cc:1918] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:37:20.938211 2800 log.cc:1196] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/be1196c6f2e9409a81c7d5c1db083212
I20260928 08:37:20.938262 2798 ts_tablet_manager.cc:1939] T be1196c6f2e9409a81c7d5c1db083212 P a5e281e87b1345cdab79e45f2ac9a89d: Deleting consensus metadata
I20260928 08:37:20.938656 2800 ts_tablet_manager.cc:1939] T be1196c6f2e9409a81c7d5c1db083212 P 8c9be510e4c9479a8c073cb54caa2d73: Deleting consensus metadata
I20260928 08:37:20.939327 2130 catalog_manager.cc:4841] TS a5e281e87b1345cdab79e45f2ac9a89d (127.0.97.1:36265): tablet be1196c6f2e9409a81c7d5c1db083212 (table test-table [id=b5b9253f598944d88f2902679447e230]) successfully deleted
I20260928 08:37:20.939608 2128 catalog_manager.cc:4841] TS 8c9be510e4c9479a8c073cb54caa2d73 (127.0.97.3:34833): tablet be1196c6f2e9409a81c7d5c1db083212 (table test-table [id=b5b9253f598944d88f2902679447e230]) successfully deleted
Sep 28 08:37:20 dist-test-slave-w6c0 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584638, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:37:20.952001 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:20.922357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60870 (local address 127.0.97.1:36265)
0928 08:37:20.929164 (+ 6807us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:20.929183 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:20.929207 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:37:20.929262 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:20.929267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:20.929353 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:20.929464 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:20.931268 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:20.932545 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:20.934415 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:20.935045 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:20.935191 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:20.946346 (+ 11155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:20.946361 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:20.946378 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:20.946413 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:20.949323 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:20.950007 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:20.950014 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:20.950017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:20.950145 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:20.950477 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:20.950479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:20.950481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:20.950922 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:20.951137 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:20.951443 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 08:37:20.951625 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6684,"thread_start_us":5989,"threads_started":1}
W20260928 08:37:20.952667 2346 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60870
W20260928 08:37:20.953338 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 2215: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:37:20.953576 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 2215
I20260928 08:37:20.978401 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 2355
Sep 28 08:37:20 dist-test-slave-w6c0 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584638, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:37:20.993465 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:20.979334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38198 (local address 127.0.97.2:41961)
0928 08:37:20.979824 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:20.979828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:20.979845 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:37:20.979897 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:20.979900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:20.979957 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:20.980054 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:20.981262 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:20.982048 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:20.983364 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:20.983777 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:20.983882 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:20.987966 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:20.987982 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:20.987998 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:20.988032 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:20.990936 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:20.991616 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:20.991624 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:20.991626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:20.991782 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:20.992106 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:20.992111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:20.992113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:20.992465 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:20.992675 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:20.992963 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 08:37:20.993145 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":271,"threads_started":1}
W20260928 08:37:20.994201 2485 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38198
W20260928 08:37:20.994899 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 2355: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:37:20.995056 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 2355
I20260928 08:37:21.019074 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 2494
Sep 28 08:37:21 dist-test-slave-w6c0 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584638, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:37:21.035091 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:21.019787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.0.97.3:34833)
0928 08:37:21.020284 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:21.020289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:21.020305 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:37:21.020357 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:21.020361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:21.020425 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:21.020515 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:21.021792 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:21.022618 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:21.023852 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:21.024279 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:21.024380 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:21.029465 (+ 5085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:21.029480 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:21.029496 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:21.029531 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:21.032587 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:21.033213 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:21.033219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:21.033221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:21.033345 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:21.033704 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:21.033708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:21.033710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:21.034093 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:21.034320 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:21.034632 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 08:37:21.034808 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":287,"threads_started":1}
W20260928 08:37:21.035784 2624 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51066
W20260928 08:37:21.036394 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 2494: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:37:21.036605 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 2494
I20260928 08:37:21.059803 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 2112
I20260928 08:37:21.060969 2184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:21.180313 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 2112
2026-09-28T08:37:21Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8287 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:37:21 dist-test-slave-w6c0 krb5kdc[2808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:37:21 dist-test-slave-w6c0 krb5kdc[2808](info): set up 2 sockets
Sep 28 08:37:21 dist-test-slave-w6c0 krb5kdc[2808](info): commencing operation
krb5kdc: starting...
W20260928 08:37:22.282836 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:37:22 dist-test-slave-w6c0 krb5kdc[2808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:37:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:37:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:37:22.483203 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44089
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:22.827193 2821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:22.828164 2821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:22.828819 2821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:22.836474 2821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:37:22.836632 2821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:22.836742 2821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:22.836845 2821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:37:22.836923 2821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:37:22.847316 2821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:22.848735 2821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:22.850059 2821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:22.863014 2829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:22.863068 2826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:22.863042 2827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:22.863531 2821 server_base.cc:1034] running on GCE node
I20260928 08:37:22.864557 2821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:22.866062 2821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:22.867311 2821 hybrid_clock.cc:648] HybridClock initialized: now 1790584642867240 us; error 54 us; skew 500 ppm
I20260928 08:37:22.867707 2821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:22 dist-test-slave-w6c0 krb5kdc[2808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:22.870891 2821 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:37:22.872375 2821 webserver.cc:460] Webserver started at http://127.0.97.62:38843/ using document root <none> and password file <none>
I20260928 08:37:22.873088 2821 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:22.873235 2821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:22.873560 2821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:22.876643 2821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "772741feb36c44c097a4d2aace6d17c6"
format_stamp: "Formatted at 2026-09-28 08:37:22 on dist-test-slave-w6c0"
server_key: "a9d5010bc71e544af28fc15926275de2"
server_key_iv: "9bec0ff53c48370ec14f5487c82c731f"
server_key_version: "encryptionkey@0"
I20260928 08:37:22.877460 2821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "772741feb36c44c097a4d2aace6d17c6"
format_stamp: "Formatted at 2026-09-28 08:37:22 on dist-test-slave-w6c0"
server_key: "a9d5010bc71e544af28fc15926275de2"
server_key_iv: "9bec0ff53c48370ec14f5487c82c731f"
server_key_version: "encryptionkey@0"
I20260928 08:37:22.882666 2821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260928 08:37:22.886135 2836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:22.887701 2821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:37:22.887923 2821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "772741feb36c44c097a4d2aace6d17c6"
format_stamp: "Formatted at 2026-09-28 08:37:22 on dist-test-slave-w6c0"
server_key: "a9d5010bc71e544af28fc15926275de2"
server_key_iv: "9bec0ff53c48370ec14f5487c82c731f"
server_key_version: "encryptionkey@0"
I20260928 08:37:22.888159 2821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:22.902899 2821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:22.907472 2821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:22.907774 2821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:22.926981 2821 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44089
I20260928 08:37:22.927004 2887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44089 every 8 connection(s)
I20260928 08:37:22.928414 2821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:37:22.932168 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 2821
I20260928 08:37:22.932402 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:37:22.933022 2888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:22.933220 388 external_mini_cluster.cc:1428] Setting key 83ff2b21ed347e60d8a5eb730c0d77c8
I20260928 08:37:22.948249 2888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6: Bootstrap starting.
Sep 28 08:37:22 dist-test-slave-w6c0 krb5kdc[2808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:22.952351 2888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:22.953469 2888 log.cc:824] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:22.956357 2888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6: No bootstrap required, opened a new log
I20260928 08:37:22.957991 2891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:22.935290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37498 (local address 127.0.97.62:44089)
0928 08:37:22.936116 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:22.936127 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:22.936171 (+ 44us) server_negotiation.cc:407] Connection header received
0928 08:37:22.938776 (+ 2605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:22.938800 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:22.939211 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:22.939567 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:22.941476 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:22.942857 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:22.944813 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:22.945542 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:22.945695 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:22.950356 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:22.950367 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:22.950390 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:22.950453 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:22.953940 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:22.954739 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:22.954746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:22.954753 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:22.954916 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:22.955331 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:22.955335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:22.955337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:22.955838 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:22.956158 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:22.956590 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 08:37:22.957029 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":369,"threads_started":1}
I20260928 08:37:22.964231 2888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772741feb36c44c097a4d2aace6d17c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44089 } }
I20260928 08:37:22.964687 2888 raft_consensus.cc:374] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:22.964804 2888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 772741feb36c44c097a4d2aace6d17c6, State: Initialized, Role: FOLLOWER
I20260928 08:37:22.965410 2888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772741feb36c44c097a4d2aace6d17c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44089 } }
I20260928 08:37:22.965619 2888 raft_consensus.cc:388] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:37:22.965714 2888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:37:22.965842 2888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:22.967998 2888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772741feb36c44c097a4d2aace6d17c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44089 } }
I20260928 08:37:22.968452 2888 leader_election.cc:302] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 772741feb36c44c097a4d2aace6d17c6; no voters:
I20260928 08:37:22.968963 2888 leader_election.cc:288] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:37:22.969120 2893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:22.969792 2893 raft_consensus.cc:686] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6 [term 1 LEADER]: Becoming Leader. State: Replica: 772741feb36c44c097a4d2aace6d17c6, State: Running, Role: LEADER
I20260928 08:37:22.970281 2893 consensus_queue.cc:237] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772741feb36c44c097a4d2aace6d17c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44089 } }
I20260928 08:37:22.971092 2888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:37:22.973592 2894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "772741feb36c44c097a4d2aace6d17c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772741feb36c44c097a4d2aace6d17c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44089 } } }
I20260928 08:37:22.973956 2894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:22.974380 2895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 772741feb36c44c097a4d2aace6d17c6. Latest consensus state: current_term: 1 leader_uuid: "772741feb36c44c097a4d2aace6d17c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772741feb36c44c097a4d2aace6d17c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44089 } } }
I20260928 08:37:22.974700 2895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:22.975967 2900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:37:22.981657 2900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:37:22.990729 2900 catalog_manager.cc:1347] Generated new cluster ID: b448302909b746cb9a98f53369c2adbb
I20260928 08:37:22.990872 2900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:37:23.007357 2900 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:37:23.008908 2900 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:37:23.019755 2900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 772741feb36c44c097a4d2aace6d17c6: Generated new TSK 0
I20260928 08:37:23.020694 2900 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:37:23.079342 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44089
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:23.428781 2914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:23.429052 2914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:23.429147 2914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:23.436333 2914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:23.436443 2914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:23.436554 2914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:37:23.446161 2914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:23.447520 2914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:23.448751 2914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:23.462324 2919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:23.462275 2920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:23.462394 2922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:23.463478 2914 server_base.cc:1034] running on GCE node
I20260928 08:37:23.464084 2914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:23.465039 2914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:23.466276 2914 hybrid_clock.cc:648] HybridClock initialized: now 1790584643466232 us; error 68 us; skew 500 ppm
I20260928 08:37:23.466650 2914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:23 dist-test-slave-w6c0 krb5kdc[2808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:23.469736 2914 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:37:23.471236 2914 webserver.cc:460] Webserver started at http://127.0.97.1:44423/ using document root <none> and password file <none>
I20260928 08:37:23.472015 2914 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:23.472157 2914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:23.472513 2914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:23.475587 2914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "b7fe96be34e34d2dbc6f9669e9062ffe"
format_stamp: "Formatted at 2026-09-28 08:37:23 on dist-test-slave-w6c0"
server_key: "73b9b3107759252d93d3d0c15b3df8b5"
server_key_iv: "2ddce74651276192b45928f62b6ab690"
server_key_version: "encryptionkey@0"
I20260928 08:37:23.476395 2914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b7fe96be34e34d2dbc6f9669e9062ffe"
format_stamp: "Formatted at 2026-09-28 08:37:23 on dist-test-slave-w6c0"
server_key: "73b9b3107759252d93d3d0c15b3df8b5"
server_key_iv: "2ddce74651276192b45928f62b6ab690"
server_key_version: "encryptionkey@0"
I20260928 08:37:23.481472 2914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:37:23.484833 2929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:23.486212 2914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:37:23.486397 2914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "b7fe96be34e34d2dbc6f9669e9062ffe"
format_stamp: "Formatted at 2026-09-28 08:37:23 on dist-test-slave-w6c0"
server_key: "73b9b3107759252d93d3d0c15b3df8b5"
server_key_iv: "2ddce74651276192b45928f62b6ab690"
server_key_version: "encryptionkey@0"
I20260928 08:37:23.486678 2914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:23.505231 2914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:23.509419 2914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:23.509719 2914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:23.510774 2914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:23.512123 2914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:23.512213 2914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:23.512387 2914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:23.512449 2914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:23.545286 2914 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33407
I20260928 08:37:23.545380 3041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33407 every 8 connection(s)
I20260928 08:37:23.546639 2914 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:37:23.549191 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 2914
I20260928 08:37:23.549412 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:37:23.550194 388 external_mini_cluster.cc:1428] Setting key 5993993a5d730f07b9f9faeb7117d29f
Sep 28 08:37:23 dist-test-slave-w6c0 krb5kdc[2808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:37:23.566643 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:23.549214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41933 (local address 127.0.97.62:44089)
0928 08:37:23.549729 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:23.549733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:23.551015 (+ 1282us) server_negotiation.cc:407] Connection header received
0928 08:37:23.552952 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:23.552957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:23.553044 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:23.553158 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:23.554616 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:23.555342 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:23.556864 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:23.557256 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:23.557394 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:23.561032 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:23.561038 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:23.561041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:23.561073 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:23.563622 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:23.564356 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:23.564367 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:23.564371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:23.564488 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:23.564926 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:23.564939 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:23.564941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:23.565189 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:23.565342 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:23.566023 (+ 681us) server_negotiation.cc:299] Negotiation successful
0928 08:37:23.566258 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":290,"threads_started":1}
I20260928 08:37:23.567639 3045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:23.549911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44089 (local address 127.0.97.1:41933)
0928 08:37:23.550809 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:23.550880 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:23.552697 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:23.553401 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:23.553410 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:23.553897 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:23.554438 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:23.554454 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:23.555500 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:23.555505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:23.556700 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:23.556712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:23.557485 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:23.557493 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:23.557721 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:23.558401 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:23.558426 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:23.560803 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:23.563768 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:23.563777 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:23.563793 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:23.564206 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:23.564600 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:23.564605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:23.564609 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:23.564796 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:23.565320 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:23.565327 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:23.565665 (+ 338us) client_negotiation.cc:769] Sending connection context
0928 08:37:23.565925 (+ 260us) client_negotiation.cc:240] Negotiation successful
0928 08:37:23.566469 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":375,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 08:37:23.569206 3042 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44089
I20260928 08:37:23.569526 3042 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:23.570253 3042 heartbeater.cc:511] Master 127.0.97.62:44089 requested a full tablet report, sending...
I20260928 08:37:23.572428 2853 ts_manager.cc:194] Registered new tserver with Master: b7fe96be34e34d2dbc6f9669e9062ffe (127.0.97.1:33407)
I20260928 08:37:23.573848 2853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:37:23.594148 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44089
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:23.918216 3048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:23.918582 3048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:23.918761 3048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:23.926066 3048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:23.926194 3048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:23.926448 3048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:37:23.936257 3048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:23.937551 3048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:23.938896 3048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:23.951802 3054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:23.951819 3053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:23.951855 3056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:23.952891 3048 server_base.cc:1034] running on GCE node
I20260928 08:37:23.953466 3048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:23.954427 3048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:23.955713 3048 hybrid_clock.cc:648] HybridClock initialized: now 1790584643955653 us; error 50 us; skew 500 ppm
I20260928 08:37:23.956064 3048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:23 dist-test-slave-w6c0 krb5kdc[2808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:23.959226 3048 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:37:23.960641 3048 webserver.cc:460] Webserver started at http://127.0.97.2:34181/ using document root <none> and password file <none>
I20260928 08:37:23.961362 3048 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:23.961503 3048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:23.961817 3048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:23.964725 3048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "e6c35859fce748229ede5e5fad340406"
format_stamp: "Formatted at 2026-09-28 08:37:23 on dist-test-slave-w6c0"
server_key: "8a9489914f918c71a0a0bb9b5c35d2cc"
server_key_iv: "d731c1b958c09b61b6701c76cf587ddb"
server_key_version: "encryptionkey@0"
I20260928 08:37:23.965497 3048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e6c35859fce748229ede5e5fad340406"
format_stamp: "Formatted at 2026-09-28 08:37:23 on dist-test-slave-w6c0"
server_key: "8a9489914f918c71a0a0bb9b5c35d2cc"
server_key_iv: "d731c1b958c09b61b6701c76cf587ddb"
server_key_version: "encryptionkey@0"
I20260928 08:37:23.970618 3048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 08:37:23.974208 3063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:23.975636 3048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:37:23.975806 3048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "e6c35859fce748229ede5e5fad340406"
format_stamp: "Formatted at 2026-09-28 08:37:23 on dist-test-slave-w6c0"
server_key: "8a9489914f918c71a0a0bb9b5c35d2cc"
server_key_iv: "d731c1b958c09b61b6701c76cf587ddb"
server_key_version: "encryptionkey@0"
I20260928 08:37:23.976011 3048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:23.997833 3048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:24.002426 3048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:24.002810 3048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:24.003813 3048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:24.005227 3048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:24.005308 3048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:24.005461 3048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:24.005512 3048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:24.039338 3048 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38819
I20260928 08:37:24.039388 3175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38819 every 8 connection(s)
I20260928 08:37:24.040599 3048 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:37:24.043479 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 3048
I20260928 08:37:24.043670 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:37:24.044265 388 external_mini_cluster.cc:1428] Setting key a0bea3bb65bba65b8a8a91b1761ff8e6
Sep 28 08:37:24 dist-test-slave-w6c0 krb5kdc[2808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 08:37:24.062369 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:24.043204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54751 (local address 127.0.97.62:44089)
0928 08:37:24.043393 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:24.043397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:24.045095 (+ 1698us) server_negotiation.cc:407] Connection header received
0928 08:37:24.047325 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:24.047330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:24.047422 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:24.047531 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:24.049159 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:24.049862 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:24.051472 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:24.051904 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:24.052041 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:24.056258 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:24.056263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:24.056267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:24.056299 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:24.059258 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:24.059997 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:24.060003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:24.060006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:24.060131 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:24.060578 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:24.060583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:24.060586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:24.060859 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:24.061010 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:24.061766 (+ 756us) server_negotiation.cc:299] Negotiation successful
0928 08:37:24.061987 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:37:24.063429 3178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:24.043880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44089 (local address 127.0.97.2:54751)
0928 08:37:24.044896 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:24.044958 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:24.047034 (+ 2076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:24.047773 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:24.047782 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:24.048446 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:24.048992 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:24.049009 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:24.050010 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:24.050022 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:24.051323 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:24.051335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:24.052134 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:24.052141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:24.052371 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:24.053080 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:24.053108 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:24.055930 (+ 2822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:24.059410 (+ 3480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:24.059419 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:24.059438 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:24.059866 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:24.060258 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:24.060265 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:24.060268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:24.060459 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:24.060986 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:24.060994 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:24.061343 (+ 349us) client_negotiation.cc:769] Sending connection context
0928 08:37:24.061667 (+ 324us) client_negotiation.cc:240] Negotiation successful
0928 08:37:24.062202 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":383,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 08:37:24.064935 3176 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44089
I20260928 08:37:24.065268 3176 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:24.066128 3176 heartbeater.cc:511] Master 127.0.97.62:44089 requested a full tablet report, sending...
I20260928 08:37:24.067860 2853 ts_manager.cc:194] Registered new tserver with Master: e6c35859fce748229ede5e5fad340406 (127.0.97.2:38819)
I20260928 08:37:24.069105 2853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:37:24.091024 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44089
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:24.419276 3181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:24.419580 3181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:24.419747 3181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:24.426867 3181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:24.427024 3181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:24.427202 3181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:37:24.436956 3181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:24.438259 3181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:24.439558 3181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:24.452441 3187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:24.452394 3189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:24.452898 3186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:24.453503 3181 server_base.cc:1034] running on GCE node
I20260928 08:37:24.454276 3181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:24.455344 3181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:24.456605 3181 hybrid_clock.cc:648] HybridClock initialized: now 1790584644456556 us; error 58 us; skew 500 ppm
I20260928 08:37:24.456949 3181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:24 dist-test-slave-w6c0 krb5kdc[2808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:24.460096 3181 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:37:24.461501 3181 webserver.cc:460] Webserver started at http://127.0.97.3:45073/ using document root <none> and password file <none>
I20260928 08:37:24.462145 3181 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:24.462244 3181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:24.462575 3181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:24.465404 3181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "86a0e05eb81747729c245f1a00a1a7e6"
format_stamp: "Formatted at 2026-09-28 08:37:24 on dist-test-slave-w6c0"
server_key: "d14e65a939bbd7a784cacc4779772e9c"
server_key_iv: "d1f8a2fd8f1854bc720ce14fd4ad0f42"
server_key_version: "encryptionkey@0"
I20260928 08:37:24.466151 3181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "86a0e05eb81747729c245f1a00a1a7e6"
format_stamp: "Formatted at 2026-09-28 08:37:24 on dist-test-slave-w6c0"
server_key: "d14e65a939bbd7a784cacc4779772e9c"
server_key_iv: "d1f8a2fd8f1854bc720ce14fd4ad0f42"
server_key_version: "encryptionkey@0"
I20260928 08:37:24.471402 3181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 08:37:24.475126 3196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:24.476794 3181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 08:37:24.476943 3181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "86a0e05eb81747729c245f1a00a1a7e6"
format_stamp: "Formatted at 2026-09-28 08:37:24 on dist-test-slave-w6c0"
server_key: "d14e65a939bbd7a784cacc4779772e9c"
server_key_iv: "d1f8a2fd8f1854bc720ce14fd4ad0f42"
server_key_version: "encryptionkey@0"
I20260928 08:37:24.477154 3181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:24.505807 3181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:24.510305 3181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:24.510651 3181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:24.511579 3181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:24.512874 3181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:24.512948 3181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:24.513029 3181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:24.513073 3181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:24.545312 3181 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35275
I20260928 08:37:24.545423 3308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35275 every 8 connection(s)
I20260928 08:37:24.546723 3181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:37:24.549830 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 3181
I20260928 08:37:24.550047 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:37:24.550694 388 external_mini_cluster.cc:1428] Setting key fb644f831391fd8daee0e66d535d04b6
Sep 28 08:37:24 dist-test-slave-w6c0 krb5kdc[2808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:24.566355 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:24.549486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48631 (local address 127.0.97.62:44089)
0928 08:37:24.549683 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:24.549689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:24.551091 (+ 1402us) server_negotiation.cc:407] Connection header received
0928 08:37:24.553291 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:24.553297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:24.553384 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:24.553504 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:24.555010 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:24.555776 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:24.557214 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:24.557595 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:24.557692 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:24.560893 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:24.560898 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:24.560901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:24.560931 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:24.563332 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:24.564120 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:24.564128 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:24.564134 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:24.564271 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:24.564687 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:24.564692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:24.564696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:24.564967 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:24.565117 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:24.565805 (+ 688us) server_negotiation.cc:299] Negotiation successful
0928 08:37:24.566013 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:37:24.567334 3311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:24.549987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44089 (local address 127.0.97.3:48631)
0928 08:37:24.550927 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:24.550976 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:24.553038 (+ 2062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:24.553731 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:24.553751 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:24.554226 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:24.554841 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:24.554859 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:24.555898 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:24.555902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:24.557068 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:24.557079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:24.557779 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:24.557784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:24.557964 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:24.558608 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:24.558629 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:24.560683 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:24.563474 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:24.563482 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:24.563498 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:24.563943 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:24.564384 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:24.564390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:24.564393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:24.564570 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:24.565097 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:24.565104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:24.565428 (+ 324us) client_negotiation.cc:769] Sending connection context
0928 08:37:24.565679 (+ 251us) client_negotiation.cc:240] Negotiation successful
0928 08:37:24.566173 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":422,"threads_started":1}
I20260928 08:37:24.568527 3309 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44089
I20260928 08:37:24.568820 3309 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:24.569471 3309 heartbeater.cc:511] Master 127.0.97.62:44089 requested a full tablet report, sending...
I20260928 08:37:24.570863 2853 ts_manager.cc:194] Registered new tserver with Master: 86a0e05eb81747729c245f1a00a1a7e6 (127.0.97.3:35275)
I20260928 08:37:24.571677 2853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48631
I20260928 08:37:24.576855 3042 heartbeater.cc:503] Master 127.0.97.62:44089 was elected leader, sending a full tablet report...
I20260928 08:37:24.578163 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 08:37:24.608420 3044 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:37:24.606226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43150 (local address 127.0.97.62:44089)
0928 08:37:24.606443 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:24.606448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:24.606890 (+ 442us) server_negotiation.cc:407] Connection header received
0928 08:37:24.607033 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:24.607037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:24.607101 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:24.607199 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:24.608239 (+ 1040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43150: BlockingRecv error: recv got EOF from 127.0.0.1:43150 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:37:24.611912 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 2914
W20260928 08:37:24.614977 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 2914: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:33407: server requires authentication, but client does not have Kerberos credentials available
I20260928 08:37:24.615151 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 2914
I20260928 08:37:24.631424 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 3048
W20260928 08:37:24.634076 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 3048: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:38819: server requires authentication, but client does not have Kerberos credentials available
I20260928 08:37:24.634246 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 3048
I20260928 08:37:24.649509 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 3181
W20260928 08:37:24.652374 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 3181: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:35275: server requires authentication, but client does not have Kerberos credentials available
I20260928 08:37:24.652520 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 3181
I20260928 08:37:24.668671 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 2821
I20260928 08:37:24.670050 2883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:24.771534 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 2821
2026-09-28T08:37:24Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3581 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:37:24 dist-test-slave-w6c0 krb5kdc[3326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:37:24 dist-test-slave-w6c0 krb5kdc[3326](info): set up 2 sockets
Sep 28 08:37:24 dist-test-slave-w6c0 krb5kdc[3326](info): commencing operation
krb5kdc: starting...
W20260928 08:37:25.862928 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:37:25 dist-test-slave-w6c0 krb5kdc[3326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:37:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:37:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:37:26.058311 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46173
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:26.384337 3339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:26.384666 3339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:26.384821 3339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:26.392133 3339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:37:26.392283 3339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:26.392400 3339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:26.392498 3339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:37:26.392589 3339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:37:26.402259 3339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:26.403649 3339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:26.405000 3339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:26.416780 3344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:26.416843 3347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:26.416821 3345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:26.417783 3339 server_base.cc:1034] running on GCE node
I20260928 08:37:26.418648 3339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:26.420058 3339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:26.421275 3339 hybrid_clock.cc:648] HybridClock initialized: now 1790584646421228 us; error 41 us; skew 500 ppm
I20260928 08:37:26.421657 3339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:26 dist-test-slave-w6c0 krb5kdc[3326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:26.429154 3339 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:37:26.430787 3339 webserver.cc:460] Webserver started at http://127.0.97.62:39109/ using document root <none> and password file <none>
I20260928 08:37:26.431483 3339 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:26.431622 3339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:26.431950 3339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:26.435060 3339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "63280a3418f4462a9d12f1d88353d92d"
format_stamp: "Formatted at 2026-09-28 08:37:26 on dist-test-slave-w6c0"
server_key: "e5662074b042adc808616079cf495e90"
server_key_iv: "ddc48401172380f3a6e53caa177cf661"
server_key_version: "encryptionkey@0"
I20260928 08:37:26.435860 3339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "63280a3418f4462a9d12f1d88353d92d"
format_stamp: "Formatted at 2026-09-28 08:37:26 on dist-test-slave-w6c0"
server_key: "e5662074b042adc808616079cf495e90"
server_key_iv: "ddc48401172380f3a6e53caa177cf661"
server_key_version: "encryptionkey@0"
I20260928 08:37:26.441022 3339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:37:26.444453 3354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:26.445853 3339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:37:26.446101 3339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "63280a3418f4462a9d12f1d88353d92d"
format_stamp: "Formatted at 2026-09-28 08:37:26 on dist-test-slave-w6c0"
server_key: "e5662074b042adc808616079cf495e90"
server_key_iv: "ddc48401172380f3a6e53caa177cf661"
server_key_version: "encryptionkey@0"
I20260928 08:37:26.446300 3339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:26.468873 3339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:26.473085 3339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:26.473392 3339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:26.491825 3339 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46173
I20260928 08:37:26.491853 3405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46173 every 8 connection(s)
I20260928 08:37:26.493172 3339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:37:26.496416 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 3339
I20260928 08:37:26.496627 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:37:26.497182 3406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:26.497465 388 external_mini_cluster.cc:1428] Setting key cf4c0a5e9a6887e2224b4a53e56374ba
I20260928 08:37:26.511857 3406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d: Bootstrap starting.
Sep 28 08:37:26 dist-test-slave-w6c0 krb5kdc[3326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:26.515793 3406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:26.517141 3406 log.cc:824] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:26.520515 3406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d: No bootstrap required, opened a new log
I20260928 08:37:26.521837 3409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:26.499347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41708 (local address 127.0.97.62:46173)
0928 08:37:26.500289 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:26.500300 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:26.500343 (+ 43us) server_negotiation.cc:407] Connection header received
0928 08:37:26.502072 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:26.502097 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:26.502477 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:26.502859 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:26.504663 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:26.505961 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:26.507735 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:26.508462 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:26.508604 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:26.513282 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:26.513294 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:26.513322 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:26.513375 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:26.517288 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:26.518097 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:26.518112 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:26.518121 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:26.518294 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:26.518728 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:26.518733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:26.518737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:26.519395 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:26.519700 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:26.520056 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 08:37:26.520485 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":430,"threads_started":1}
I20260928 08:37:26.525797 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:37:26.529064 3406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63280a3418f4462a9d12f1d88353d92d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46173 } }
I20260928 08:37:26.529433 3406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:26.529541 3406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63280a3418f4462a9d12f1d88353d92d, State: Initialized, Role: FOLLOWER
I20260928 08:37:26.530154 3406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63280a3418f4462a9d12f1d88353d92d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46173 } }
I20260928 08:37:26.530387 3406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:37:26.530544 3406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:37:26.530736 3406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:26.534070 3406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63280a3418f4462a9d12f1d88353d92d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46173 } }
I20260928 08:37:26.534655 3406 leader_election.cc:302] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63280a3418f4462a9d12f1d88353d92d; no voters:
I20260928 08:37:26.535297 3406 leader_election.cc:288] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:37:26.535408 3412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:26.536671 3412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d [term 1 LEADER]: Becoming Leader. State: Replica: 63280a3418f4462a9d12f1d88353d92d, State: Running, Role: LEADER
I20260928 08:37:26.537194 3412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63280a3418f4462a9d12f1d88353d92d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46173 } }
I20260928 08:37:26.537523 3406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:37:26.539794 3414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63280a3418f4462a9d12f1d88353d92d. Latest consensus state: current_term: 1 leader_uuid: "63280a3418f4462a9d12f1d88353d92d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63280a3418f4462a9d12f1d88353d92d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46173 } } }
I20260928 08:37:26.540138 3414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:26.540689 3413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63280a3418f4462a9d12f1d88353d92d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63280a3418f4462a9d12f1d88353d92d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46173 } } }
I20260928 08:37:26.540990 3413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:26.549300 3418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:37:26.559561 3418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:37:26.577702 3418 catalog_manager.cc:1347] Generated new cluster ID: 7aa405d9ecb04a89bb91ccd4d6c5aef2
I20260928 08:37:26.577860 3418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:37:26.594831 3418 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:37:26.595958 3409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:26.586788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41722 (local address 127.0.97.62:46173)
0928 08:37:26.586949 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:26.586955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:26.587482 (+ 527us) server_negotiation.cc:407] Connection header received
0928 08:37:26.587650 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:26.587656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:26.587746 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:26.587873 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:26.588992 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:26.590179 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:26.592016 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:26.592670 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:26.592805 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:26.594007 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:26.594014 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:26.594019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:37:26.594922 (+ 903us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:26.594981 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:37:26.595007 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:26.595135 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:26.595296 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 08:37:26.595399 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":52}
I20260928 08:37:26.596494 3418 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:37:26.610916 3418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63280a3418f4462a9d12f1d88353d92d: Generated new TSK 0
I20260928 08:37:26.612120 3418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:37:26.624392 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 3339
I20260928 08:37:26.625833 3401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:26.715474 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 3339
2026-09-28T08:37:26Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1935 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:37:26 dist-test-slave-w6c0 krb5kdc[3440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:37:26 dist-test-slave-w6c0 krb5kdc[3440](info): set up 2 sockets
Sep 28 08:37:26 dist-test-slave-w6c0 krb5kdc[3440](info): commencing operation
krb5kdc: starting...
W20260928 08:37:27.798867 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:37:27 dist-test-slave-w6c0 krb5kdc[3440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:37:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:37:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:37:27.993896 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44091
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:28.315258 3453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:28.315557 3453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:28.315742 3453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:28.322968 3453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:37:28.323081 3453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:28.323199 3453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:28.323307 3453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:37:28.323374 3453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:37:28.333127 3453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:28.334519 3453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:28.335839 3453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:28.347858 3458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:28.347879 3459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:28.347931 3461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:28.348749 3453 server_base.cc:1034] running on GCE node
I20260928 08:37:28.349635 3453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:28.351202 3453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:28.352468 3453 hybrid_clock.cc:648] HybridClock initialized: now 1790584648352419 us; error 35 us; skew 500 ppm
I20260928 08:37:28.352849 3453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:28 dist-test-slave-w6c0 krb5kdc[3440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:28.356292 3453 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:37:28.357890 3453 webserver.cc:460] Webserver started at http://127.0.97.62:37723/ using document root <none> and password file <none>
I20260928 08:37:28.358678 3453 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:28.358826 3453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:28.359159 3453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:28.362135 3453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "3584fa1683274aecb8ac0686e91a4a22"
format_stamp: "Formatted at 2026-09-28 08:37:28 on dist-test-slave-w6c0"
server_key: "b034af456f66f06b3b75556a14437c8a"
server_key_iv: "b7785164892a1f7a298d6d86106dadb0"
server_key_version: "encryptionkey@0"
I20260928 08:37:28.362926 3453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "3584fa1683274aecb8ac0686e91a4a22"
format_stamp: "Formatted at 2026-09-28 08:37:28 on dist-test-slave-w6c0"
server_key: "b034af456f66f06b3b75556a14437c8a"
server_key_iv: "b7785164892a1f7a298d6d86106dadb0"
server_key_version: "encryptionkey@0"
I20260928 08:37:28.368032 3453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 08:37:28.371484 3468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:28.372934 3453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:37:28.373142 3453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "3584fa1683274aecb8ac0686e91a4a22"
format_stamp: "Formatted at 2026-09-28 08:37:28 on dist-test-slave-w6c0"
server_key: "b034af456f66f06b3b75556a14437c8a"
server_key_iv: "b7785164892a1f7a298d6d86106dadb0"
server_key_version: "encryptionkey@0"
I20260928 08:37:28.373346 3453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:28.385771 3453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:28.390096 3453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:28.390398 3453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:28.409051 3453 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44091
I20260928 08:37:28.409080 3519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44091 every 8 connection(s)
I20260928 08:37:28.410425 3453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:37:28.412087 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 3453
I20260928 08:37:28.412305 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:37:28.413106 388 external_mini_cluster.cc:1428] Setting key 9a1e856f454cda41115f7f403e6956a0
I20260928 08:37:28.415226 3520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:28.428423 3520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22: Bootstrap starting.
Sep 28 08:37:28 dist-test-slave-w6c0 krb5kdc[3440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:28.431991 3520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:28.433085 3520 log.cc:824] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:28.436313 3520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22: No bootstrap required, opened a new log
I20260928 08:37:28.439810 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:28.415081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51824 (local address 127.0.97.62:44091)
0928 08:37:28.419100 (+ 4019us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:28.419112 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:28.419157 (+ 45us) server_negotiation.cc:407] Connection header received
0928 08:37:28.420712 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:28.420734 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:28.421146 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:28.421496 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:28.423235 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:28.424524 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:28.426323 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:28.427082 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:28.427225 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:28.431720 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:28.431729 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:28.431743 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:28.431783 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:28.435365 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:28.436041 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:28.436050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:28.436058 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:28.436255 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:28.436673 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:28.436677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:28.436680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:28.437374 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:28.437679 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:28.437981 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 08:37:28.438414 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3772,"thread_start_us":3568,"threads_started":1}
I20260928 08:37:28.444764 3520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3584fa1683274aecb8ac0686e91a4a22" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44091 } }
I20260928 08:37:28.445099 3520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:28.445199 3520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3584fa1683274aecb8ac0686e91a4a22, State: Initialized, Role: FOLLOWER
I20260928 08:37:28.445775 3520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3584fa1683274aecb8ac0686e91a4a22" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44091 } }
I20260928 08:37:28.446019 3520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:37:28.446162 3520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:37:28.446336 3520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:28.448472 3520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3584fa1683274aecb8ac0686e91a4a22" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44091 } }
I20260928 08:37:28.448949 3520 leader_election.cc:302] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3584fa1683274aecb8ac0686e91a4a22; no voters:
I20260928 08:37:28.449625 3520 leader_election.cc:288] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:37:28.449764 3525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:28.450609 3525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22 [term 1 LEADER]: Becoming Leader. State: Replica: 3584fa1683274aecb8ac0686e91a4a22, State: Running, Role: LEADER
I20260928 08:37:28.451108 3525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3584fa1683274aecb8ac0686e91a4a22" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44091 } }
I20260928 08:37:28.451674 3520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:37:28.454322 3527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3584fa1683274aecb8ac0686e91a4a22. Latest consensus state: current_term: 1 leader_uuid: "3584fa1683274aecb8ac0686e91a4a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3584fa1683274aecb8ac0686e91a4a22" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44091 } } }
I20260928 08:37:28.454702 3527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:28.455070 3526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3584fa1683274aecb8ac0686e91a4a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3584fa1683274aecb8ac0686e91a4a22" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44091 } } }
I20260928 08:37:28.455367 3526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:28.457110 3533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:37:28.461440 3533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:37:28.469436 3533 catalog_manager.cc:1347] Generated new cluster ID: 51ec169eefbf4a05b2ab502603e8ce90
I20260928 08:37:28.469543 3533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:37:28.491080 3533 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:37:28.492381 3533 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:37:28.501986 3533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3584fa1683274aecb8ac0686e91a4a22: Generated new TSK 0
I20260928 08:37:28.502920 3533 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:37:28.554553 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44091
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:28.878957 3546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:28.879247 3546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:28.879410 3546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:28.886730 3546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:28.886842 3546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:28.887058 3546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:37:28.896845 3546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:28.898195 3546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:28.899619 3546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:28.913403 3554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:28.913517 3551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:28.913543 3552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:28.914278 3546 server_base.cc:1034] running on GCE node
I20260928 08:37:28.914947 3546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:28.915911 3546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:28.917137 3546 hybrid_clock.cc:648] HybridClock initialized: now 1790584648917103 us; error 54 us; skew 500 ppm
I20260928 08:37:28.917451 3546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:28 dist-test-slave-w6c0 krb5kdc[3440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:28.920588 3546 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:37:28.921970 3546 webserver.cc:460] Webserver started at http://127.0.97.1:32939/ using document root <none> and password file <none>
I20260928 08:37:28.922685 3546 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:28.922786 3546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:28.923028 3546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:28.925782 3546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "978a6a793f7842a4bff78093dc28bfde"
format_stamp: "Formatted at 2026-09-28 08:37:28 on dist-test-slave-w6c0"
server_key: "b61fbbaf61a6e866c688ef1e421f35d1"
server_key_iv: "a58383fd63fdb55ff5058959eb4adb8f"
server_key_version: "encryptionkey@0"
I20260928 08:37:28.926553 3546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "978a6a793f7842a4bff78093dc28bfde"
format_stamp: "Formatted at 2026-09-28 08:37:28 on dist-test-slave-w6c0"
server_key: "b61fbbaf61a6e866c688ef1e421f35d1"
server_key_iv: "a58383fd63fdb55ff5058959eb4adb8f"
server_key_version: "encryptionkey@0"
I20260928 08:37:28.931353 3546 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 08:37:28.934598 3561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:28.936081 3546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:37:28.936249 3546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "978a6a793f7842a4bff78093dc28bfde"
format_stamp: "Formatted at 2026-09-28 08:37:28 on dist-test-slave-w6c0"
server_key: "b61fbbaf61a6e866c688ef1e421f35d1"
server_key_iv: "a58383fd63fdb55ff5058959eb4adb8f"
server_key_version: "encryptionkey@0"
I20260928 08:37:28.936451 3546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:28.966320 3546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:28.970398 3546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:28.970726 3546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:28.971680 3546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:28.973022 3546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:28.973127 3546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:28.973277 3546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:28.973356 3546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:29.006388 3546 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41301
I20260928 08:37:29.006445 3673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41301 every 8 connection(s)
I20260928 08:37:29.007697 3546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:37:29.013821 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 3546
I20260928 08:37:29.014040 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:37:29.014746 388 external_mini_cluster.cc:1428] Setting key 9c3591854b8cc24ceca2c53468351ffb
Sep 28 08:37:29 dist-test-slave-w6c0 krb5kdc[3440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:29.030061 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:29.010208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51847 (local address 127.0.97.62:44091)
0928 08:37:29.010675 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:29.010680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:29.011501 (+ 821us) server_negotiation.cc:407] Connection header received
0928 08:37:29.013424 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:29.013430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:29.013518 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:29.013637 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:29.015282 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:29.016236 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:29.017863 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:29.018669 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:29.018822 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:29.022696 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:29.022702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:29.022705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:29.022738 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:29.026450 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:29.027274 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:29.027282 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:29.027288 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:29.027449 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:29.027865 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:29.027873 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:29.027879 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:29.028248 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:29.028411 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:29.029283 (+ 872us) server_negotiation.cc:299] Negotiation successful
0928 08:37:29.029571 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":251,"threads_started":1}
I20260928 08:37:29.030787 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:29.010678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44091 (local address 127.0.97.1:51847)
0928 08:37:29.011354 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:29.011391 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:29.013149 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:29.013869 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:29.013878 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:29.014456 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:29.015117 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:29.015134 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:29.016436 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:29.016441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:29.017699 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:29.017711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:29.018901 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:29.018907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:29.019117 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:29.019889 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:29.019913 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:29.022423 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:29.026657 (+ 4234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:29.026664 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:29.026680 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:29.027131 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:29.027562 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:29.027565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:29.027567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:29.027747 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:29.028469 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:29.028476 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:29.028873 (+ 397us) client_negotiation.cc:769] Sending connection context
0928 08:37:29.029118 (+ 245us) client_negotiation.cc:240] Negotiation successful
0928 08:37:29.029540 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":283,"threads_started":1}
I20260928 08:37:29.032449 3674 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44091
I20260928 08:37:29.032795 3674 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:29.033726 3674 heartbeater.cc:511] Master 127.0.97.62:44091 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:37:29.036118 3485 ts_manager.cc:194] Registered new tserver with Master: 978a6a793f7842a4bff78093dc28bfde (127.0.97.1:41301)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 08:37:29.037689 3485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:37:29.063832 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44091
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:29.389950 3680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:29.390233 3680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:29.390331 3680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:29.397612 3680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:29.397732 3680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:29.397847 3680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:37:29.407560 3680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:29.409091 3680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:29.410800 3680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:29.425527 3688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:29.425984 3685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:29.425509 3686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:29.426429 3680 server_base.cc:1034] running on GCE node
I20260928 08:37:29.427024 3680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:29.427971 3680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:29.429230 3680 hybrid_clock.cc:648] HybridClock initialized: now 1790584649429171 us; error 49 us; skew 500 ppm
I20260928 08:37:29.429652 3680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:29 dist-test-slave-w6c0 krb5kdc[3440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:29.432893 3680 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:37:29.434345 3680 webserver.cc:460] Webserver started at http://127.0.97.2:36991/ using document root <none> and password file <none>
I20260928 08:37:29.435101 3680 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:29.435245 3680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:29.435554 3680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:29.438421 3680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "aab8b9d977cf4b1a9bffab730151249e"
format_stamp: "Formatted at 2026-09-28 08:37:29 on dist-test-slave-w6c0"
server_key: "e35e9e7617348ce662da07fc6ab183c2"
server_key_iv: "63d0360a47a1eddffc2a97728257803f"
server_key_version: "encryptionkey@0"
I20260928 08:37:29.439268 3680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "aab8b9d977cf4b1a9bffab730151249e"
format_stamp: "Formatted at 2026-09-28 08:37:29 on dist-test-slave-w6c0"
server_key: "e35e9e7617348ce662da07fc6ab183c2"
server_key_iv: "63d0360a47a1eddffc2a97728257803f"
server_key_version: "encryptionkey@0"
I20260928 08:37:29.444576 3680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:37:29.448148 3695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:29.449406 3680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:37:29.449575 3680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "aab8b9d977cf4b1a9bffab730151249e"
format_stamp: "Formatted at 2026-09-28 08:37:29 on dist-test-slave-w6c0"
server_key: "e35e9e7617348ce662da07fc6ab183c2"
server_key_iv: "63d0360a47a1eddffc2a97728257803f"
server_key_version: "encryptionkey@0"
I20260928 08:37:29.449791 3680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:29.472968 3680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:29.477345 3680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:29.477648 3680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:29.478798 3680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:29.480162 3680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:29.480268 3680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:29.480420 3680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:29.480494 3680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:29.513662 3680 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40525
I20260928 08:37:29.513751 3807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40525 every 8 connection(s)
I20260928 08:37:29.515029 3680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:37:29.522272 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 3680
I20260928 08:37:29.522435 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:37:29.522904 388 external_mini_cluster.cc:1428] Setting key c974b45c3d1ea6cc48f02dd6409ba9e8
Sep 28 08:37:29 dist-test-slave-w6c0 krb5kdc[3440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:29.535336 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:29.517536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50425 (local address 127.0.97.62:44091)
0928 08:37:29.517736 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:29.517745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:29.519016 (+ 1271us) server_negotiation.cc:407] Connection header received
0928 08:37:29.520669 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:29.520673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:29.520739 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:29.520829 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:29.522236 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:29.523146 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:29.524957 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:29.525372 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:29.525509 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:29.529182 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:29.529192 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:29.529200 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:29.529246 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:29.532119 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:29.533044 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:29.533048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:29.533051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:29.533171 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:29.533621 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:29.533623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:29.533625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:29.533884 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:29.533990 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:29.534923 (+ 933us) server_negotiation.cc:299] Negotiation successful
0928 08:37:29.535093 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:37:29.536926 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:29.518113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44091 (local address 127.0.97.2:50425)
0928 08:37:29.518874 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:29.518910 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:29.520441 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:29.521023 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:29.521030 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:29.521526 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:29.522057 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:29.522074 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:29.523300 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:29.523305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:29.524814 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:29.524826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:29.525589 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:29.525596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:29.525835 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:29.526577 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:29.526599 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:29.528956 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:29.532263 (+ 3307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:29.532274 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:29.532292 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:29.532929 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:29.533276 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:29.533283 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:29.533290 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:29.533520 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:29.534014 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:29.534025 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:29.534470 (+ 445us) client_negotiation.cc:769] Sending connection context
0928 08:37:29.534840 (+ 370us) client_negotiation.cc:240] Negotiation successful
0928 08:37:29.535415 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":309,"threads_started":1}
I20260928 08:37:29.538503 3808 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44091
I20260928 08:37:29.538857 3808 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:29.539631 3808 heartbeater.cc:511] Master 127.0.97.62:44091 requested a full tablet report, sending...
I20260928 08:37:29.541257 3485 ts_manager.cc:194] Registered new tserver with Master: aab8b9d977cf4b1a9bffab730151249e (127.0.97.2:40525)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 08:37:29.542486 3485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:37:29.570109 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44091
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:29.899791 3813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:29.900061 3813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:29.900209 3813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:29.907644 3813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:29.907756 3813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:29.907967 3813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:37:29.918080 3813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:29.919513 3813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:29.920843 3813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:29.933991 3818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:29.933916 3821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:29.933961 3819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:29.934733 3813 server_base.cc:1034] running on GCE node
I20260928 08:37:29.935377 3813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:29.936345 3813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:29.937547 3813 hybrid_clock.cc:648] HybridClock initialized: now 1790584649937489 us; error 54 us; skew 500 ppm
I20260928 08:37:29.937875 3813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:29 dist-test-slave-w6c0 krb5kdc[3440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:29.941015 3813 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:37:29.942482 3813 webserver.cc:460] Webserver started at http://127.0.97.3:45567/ using document root <none> and password file <none>
I20260928 08:37:29.943244 3813 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:29.943370 3813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:29.943686 3813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:29.946798 3813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "45c50fcdc1514d31a9507a235d6ab304"
format_stamp: "Formatted at 2026-09-28 08:37:29 on dist-test-slave-w6c0"
server_key: "eeee5ad2db6f916067050c059a6148a0"
server_key_iv: "f65e4460c4bfa45470b31edb7915c626"
server_key_version: "encryptionkey@0"
I20260928 08:37:29.947636 3813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "45c50fcdc1514d31a9507a235d6ab304"
format_stamp: "Formatted at 2026-09-28 08:37:29 on dist-test-slave-w6c0"
server_key: "eeee5ad2db6f916067050c059a6148a0"
server_key_iv: "f65e4460c4bfa45470b31edb7915c626"
server_key_version: "encryptionkey@0"
I20260928 08:37:29.952829 3813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 08:37:29.956568 3828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:29.958051 3813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 08:37:29.958240 3813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "45c50fcdc1514d31a9507a235d6ab304"
format_stamp: "Formatted at 2026-09-28 08:37:29 on dist-test-slave-w6c0"
server_key: "eeee5ad2db6f916067050c059a6148a0"
server_key_iv: "f65e4460c4bfa45470b31edb7915c626"
server_key_version: "encryptionkey@0"
I20260928 08:37:29.958441 3813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:29.974624 3813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:29.978883 3813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:29.979177 3813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:29.980104 3813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:29.981458 3813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:29.981528 3813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:29.981701 3813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:29.981752 3813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:30.014278 3813 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40039
I20260928 08:37:30.014350 3940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40039 every 8 connection(s)
I20260928 08:37:30.015633 3813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:37:30.019032 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 3813
I20260928 08:37:30.019261 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:37:30.019874 388 external_mini_cluster.cc:1428] Setting key c4c470f8f145bb4a4d2f262fb04b628a
Sep 28 08:37:30 dist-test-slave-w6c0 krb5kdc[3440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:30.037039 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:30.018317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48485 (local address 127.0.97.62:44091)
0928 08:37:30.018590 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:30.018602 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:30.019899 (+ 1297us) server_negotiation.cc:407] Connection header received
0928 08:37:30.022251 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:30.022256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:30.022337 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:30.022448 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:30.024185 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:30.024964 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:30.026575 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:30.026984 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:30.027084 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:30.030661 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:30.030686 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:30.030689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:30.030722 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:30.033387 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:30.034412 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:30.034417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:30.034421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:30.034636 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:30.035205 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:30.035209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:30.035212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:30.035581 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:30.035735 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:30.036571 (+ 836us) server_negotiation.cc:299] Negotiation successful
0928 08:37:30.036787 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:37:30.038343 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:30.018835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44091 (local address 127.0.97.3:48485)
0928 08:37:30.019714 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:30.019759 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:30.021957 (+ 2198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:30.022728 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:30.022738 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:30.023423 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:30.024013 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:30.024030 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:30.025104 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:30.025108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:30.026375 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:30.026387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:30.027212 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:30.027218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:30.027417 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:30.028130 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:30.028155 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:30.030408 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:30.033625 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:30.033634 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:30.033652 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:30.034247 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:30.034777 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:30.034782 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:30.034785 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:30.035037 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:30.035748 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:30.035756 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:30.036212 (+ 456us) client_negotiation.cc:769] Sending connection context
0928 08:37:30.036448 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 08:37:30.036847 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":336,"threads_started":1}
I20260928 08:37:30.039847 3941 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44091
I20260928 08:37:30.040132 3941 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:30.040918 3941 heartbeater.cc:511] Master 127.0.97.62:44091 requested a full tablet report, sending...
I20260928 08:37:30.040948 3674 heartbeater.cc:503] Master 127.0.97.62:44091 was elected leader, sending a full tablet report...
I20260928 08:37:30.042466 3484 ts_manager.cc:194] Registered new tserver with Master: 45c50fcdc1514d31a9507a235d6ab304 (127.0.97.3:40039)
I20260928 08:37:30.043349 3484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48485
I20260928 08:37:30.048483 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:37:30 dist-test-slave-w6c0 krb5kdc[3440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 08:37:30 dist-test-slave-w6c0 krb5kdc[3440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:30.098095 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:30.084100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51840 (local address 127.0.97.62:44091)
0928 08:37:30.084346 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:30.084355 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:30.084748 (+ 393us) server_negotiation.cc:407] Connection header received
0928 08:37:30.084964 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:30.084970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:30.085044 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:30.085154 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:30.086577 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:30.087312 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:30.088559 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:30.088987 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:30.089126 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:30.093218 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:30.093223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:30.093226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:30.093263 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:30.095827 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:30.096428 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:30.096432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:30.096434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:30.096546 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:30.096863 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:30.096866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:30.096868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:30.097162 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:30.097333 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:30.097643 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 08:37:30.097820 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 08:37:30.098707 3484 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51840
I20260928 08:37:30.100247 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 3546
Sep 28 08:37:30 dist-test-slave-w6c0 krb5kdc[3440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:37:30.115665 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:30.101273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.0.97.1:41301)
0928 08:37:30.101901 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:30.101908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:30.101933 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:37:30.101985 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:30.101990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:30.102199 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:30.102326 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:30.103527 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:30.104518 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:30.105729 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:30.106267 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:30.106368 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:30.110208 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:30.110219 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:30.110233 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:30.110262 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:30.112903 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:30.113518 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:30.113525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:30.113527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:30.113640 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:30.114024 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:30.114027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:30.114030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:30.114380 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:30.114746 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:30.115061 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 08:37:30.115325 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":308,"threads_started":1}
W20260928 08:37:30.116894 3669 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59930
W20260928 08:37:30.117926 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 3546: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:37:30.118108 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 3546
I20260928 08:37:30.134051 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 3680
Sep 28 08:37:30 dist-test-slave-w6c0 krb5kdc[3440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:37:30.151767 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:30.134882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55404 (local address 127.0.97.2:40525)
0928 08:37:30.135446 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:30.135453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:30.135480 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:37:30.135541 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:30.135548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:30.135770 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:30.135899 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:30.137710 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:30.139866 (+ 2156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:30.141100 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:30.141642 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:30.141792 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:30.146060 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:30.146071 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:30.146086 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:30.146118 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:30.148841 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:30.149514 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:30.149522 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:30.149524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:30.149647 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:30.150027 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:30.150033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:30.150036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:30.150367 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:30.150702 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:30.151100 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 08:37:30.151364 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":312,"threads_started":1}
W20260928 08:37:30.153003 3803 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55404
W20260928 08:37:30.154081 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 3680: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:37:30.154242 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 3680
I20260928 08:37:30.170320 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 3813
Sep 28 08:37:30 dist-test-slave-w6c0 krb5kdc[3440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:37:30.186866 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:30.171341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42410 (local address 127.0.97.3:40039)
0928 08:37:30.171874 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:30.171881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:30.171910 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:37:30.171969 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:30.171976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:30.172197 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:30.172369 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:30.173723 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:30.174867 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:30.176096 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:30.176675 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:30.176816 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:30.181122 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:30.181133 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:30.181147 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:30.181187 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:30.184026 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:30.184723 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:30.184730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:30.184732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:30.184860 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:30.185224 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:30.185231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:30.185234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:30.185573 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:30.185918 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:30.186218 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 08:37:30.186497 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":247,"threads_started":1}
W20260928 08:37:30.188169 3936 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42410
W20260928 08:37:30.189226 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 3813: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:37:30.189370 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 3813
I20260928 08:37:30.205111 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 3453
I20260928 08:37:30.206254 3515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:30.308669 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 3453
2026-09-28T08:37:30Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3602 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:37:30 dist-test-slave-w6c0 krb5kdc[3958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:37:30 dist-test-slave-w6c0 krb5kdc[3958](info): set up 2 sockets
Sep 28 08:37:30 dist-test-slave-w6c0 krb5kdc[3958](info): commencing operation
krb5kdc: starting...
W20260928 08:37:31.398993 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:37:31 dist-test-slave-w6c0 krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:37:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:37:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:37:31.598067 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36885
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:31.924943 3971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:31.925283 3971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:31.925451 3971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:31.932853 3971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:37:31.932963 3971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:31.933044 3971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:31.933144 3971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:37:31.933225 3971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:37:31.943311 3971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:31.944679 3971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:31.946095 3971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:31.958263 3979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:31.958359 3977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:31.958312 3976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:31.959219 3971 server_base.cc:1034] running on GCE node
I20260928 08:37:31.960016 3971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:31.961551 3971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:31.962779 3971 hybrid_clock.cc:648] HybridClock initialized: now 1790584651962742 us; error 28 us; skew 500 ppm
I20260928 08:37:31.963114 3971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:31 dist-test-slave-w6c0 krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:31.971997 3971 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:37:31.973562 3971 webserver.cc:460] Webserver started at http://127.0.97.62:45463/ using document root <none> and password file <none>
I20260928 08:37:31.974220 3971 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:31.974323 3971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:31.974664 3971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:31.977653 3971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "ed630aa5434d41809dcdee285d237194"
format_stamp: "Formatted at 2026-09-28 08:37:31 on dist-test-slave-w6c0"
server_key: "4356dc8dbbe1c7f3669fcc8b62c5fd47"
server_key_iv: "bf7233a04d7d0442716ddf73cd404ce4"
server_key_version: "encryptionkey@0"
I20260928 08:37:31.978366 3971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "ed630aa5434d41809dcdee285d237194"
format_stamp: "Formatted at 2026-09-28 08:37:31 on dist-test-slave-w6c0"
server_key: "4356dc8dbbe1c7f3669fcc8b62c5fd47"
server_key_iv: "bf7233a04d7d0442716ddf73cd404ce4"
server_key_version: "encryptionkey@0"
I20260928 08:37:31.983608 3971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:37:31.987010 3986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:31.988261 3971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:37:31.988427 3971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "ed630aa5434d41809dcdee285d237194"
format_stamp: "Formatted at 2026-09-28 08:37:31 on dist-test-slave-w6c0"
server_key: "4356dc8dbbe1c7f3669fcc8b62c5fd47"
server_key_iv: "bf7233a04d7d0442716ddf73cd404ce4"
server_key_version: "encryptionkey@0"
I20260928 08:37:31.988600 3971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:32.017246 3971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:32.021679 3971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:32.022002 3971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:32.040962 3971 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36885
I20260928 08:37:32.041002 4037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36885 every 8 connection(s)
I20260928 08:37:32.042346 3971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:37:32.046002 4038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:32.047461 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 3971
I20260928 08:37:32.047688 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:37:32.048491 388 external_mini_cluster.cc:1428] Setting key 697cf6a791cbedd94cb5e6a148efd76d
I20260928 08:37:32.060631 4038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194: Bootstrap starting.
Sep 28 08:37:32 dist-test-slave-w6c0 krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:32.064404 4038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:32.065784 4038 log.cc:824] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:32.069069 4038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194: No bootstrap required, opened a new log
I20260928 08:37:32.072139 4041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:32.050636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.0.97.62:36885)
0928 08:37:32.051482 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:32.051493 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:32.051539 (+ 46us) server_negotiation.cc:407] Connection header received
0928 08:37:32.053078 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:32.053106 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:32.053509 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:32.053837 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:32.055650 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:32.056990 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:32.058805 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:32.059477 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:32.059614 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:32.064114 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:32.064125 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:32.064146 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:32.064194 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:32.067822 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:32.068466 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:32.068478 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:32.068488 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:32.068676 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:32.069025 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:32.069033 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:32.069038 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:32.069724 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:32.070047 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:32.070356 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 08:37:32.070777 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":635,"thread_start_us":375,"threads_started":1}
I20260928 08:37:32.078068 4038 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed630aa5434d41809dcdee285d237194" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36885 } }
I20260928 08:37:32.078372 4038 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:32.078440 4038 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed630aa5434d41809dcdee285d237194, State: Initialized, Role: FOLLOWER
I20260928 08:37:32.078962 4038 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed630aa5434d41809dcdee285d237194" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36885 } }
I20260928 08:37:32.079164 4038 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:37:32.079254 4038 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:37:32.079420 4038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:32.081526 4038 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed630aa5434d41809dcdee285d237194" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36885 } }
I20260928 08:37:32.082005 4038 leader_election.cc:302] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed630aa5434d41809dcdee285d237194; no voters:
I20260928 08:37:32.082620 4038 leader_election.cc:288] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:37:32.082731 4043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:32.083415 4043 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194 [term 1 LEADER]: Becoming Leader. State: Replica: ed630aa5434d41809dcdee285d237194, State: Running, Role: LEADER
I20260928 08:37:32.083894 4043 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed630aa5434d41809dcdee285d237194" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36885 } }
I20260928 08:37:32.084666 4038 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:37:32.086782 4045 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed630aa5434d41809dcdee285d237194. Latest consensus state: current_term: 1 leader_uuid: "ed630aa5434d41809dcdee285d237194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed630aa5434d41809dcdee285d237194" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36885 } } }
I20260928 08:37:32.087265 4045 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:32.087176 4044 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed630aa5434d41809dcdee285d237194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed630aa5434d41809dcdee285d237194" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36885 } } }
I20260928 08:37:32.087932 4044 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:32.090049 4050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:37:32.095185 4050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:37:32.104488 4050 catalog_manager.cc:1347] Generated new cluster ID: 84234568abcd4b928d1df5eb702be602
I20260928 08:37:32.104660 4050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:37:32.139966 4050 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:37:32.141328 4050 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:37:32.158658 4050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed630aa5434d41809dcdee285d237194: Generated new TSK 0
I20260928 08:37:32.159600 4050 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:37:32.206080 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36885
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:32.533937 4064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:32.534216 4064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:32.534312 4064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:32.541545 4064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:32.541654 4064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:32.541767 4064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:37:32.551517 4064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:32.552726 4064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:32.554105 4064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:32.567494 4069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:32.567589 4072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:32.567557 4070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:32.568341 4064 server_base.cc:1034] running on GCE node
I20260928 08:37:32.568921 4064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:32.569814 4064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:32.571038 4064 hybrid_clock.cc:648] HybridClock initialized: now 1790584652570988 us; error 31 us; skew 500 ppm
I20260928 08:37:32.571367 4064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:32 dist-test-slave-w6c0 krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:32.580405 4064 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:37:32.581846 4064 webserver.cc:460] Webserver started at http://127.0.97.1:36879/ using document root <none> and password file <none>
I20260928 08:37:32.582615 4064 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:32.582760 4064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:32.583074 4064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:32.586139 4064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "e99d81a30f2b4c91b6c18f6d68539b09"
format_stamp: "Formatted at 2026-09-28 08:37:32 on dist-test-slave-w6c0"
server_key: "2e58a50fae0b3bb7070627c71ac2b85d"
server_key_iv: "8ce397078d9592739a26f7536c873b96"
server_key_version: "encryptionkey@0"
I20260928 08:37:32.586956 4064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e99d81a30f2b4c91b6c18f6d68539b09"
format_stamp: "Formatted at 2026-09-28 08:37:32 on dist-test-slave-w6c0"
server_key: "2e58a50fae0b3bb7070627c71ac2b85d"
server_key_iv: "8ce397078d9592739a26f7536c873b96"
server_key_version: "encryptionkey@0"
I20260928 08:37:32.592240 4064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:37:32.595952 4079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:32.597295 4064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 08:37:32.597529 4064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "e99d81a30f2b4c91b6c18f6d68539b09"
format_stamp: "Formatted at 2026-09-28 08:37:32 on dist-test-slave-w6c0"
server_key: "2e58a50fae0b3bb7070627c71ac2b85d"
server_key_iv: "8ce397078d9592739a26f7536c873b96"
server_key_version: "encryptionkey@0"
I20260928 08:37:32.597754 4064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:32.616055 4064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:32.620172 4064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:32.620549 4064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:32.621611 4064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:32.623093 4064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:32.623202 4064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:32.623356 4064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:32.623411 4064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:32.657032 4064 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38345
I20260928 08:37:32.657081 4191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38345 every 8 connection(s)
I20260928 08:37:32.658463 4064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:37:32.665122 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 4064
I20260928 08:37:32.665360 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:37:32.665993 388 external_mini_cluster.cc:1428] Setting key 04728f258421119d2d2c0ded30e89277
Sep 28 08:37:32 dist-test-slave-w6c0 krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:32.679824 4194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:32.661203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50247 (local address 127.0.97.62:36885)
0928 08:37:32.661657 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:32.661662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:32.662646 (+ 984us) server_negotiation.cc:407] Connection header received
0928 08:37:32.664451 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:32.664459 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:32.664544 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:32.664665 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:32.666368 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:32.667301 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:32.669022 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:32.669432 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:32.669580 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:32.673077 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:32.673083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:32.673086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:32.673118 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:32.676467 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:32.677207 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:32.677216 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:32.677223 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:32.677364 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:32.677770 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:32.677777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:32.677784 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:32.678175 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:32.678339 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:32.679075 (+ 736us) server_negotiation.cc:299] Negotiation successful
0928 08:37:32.679335 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":232,"threads_started":1}
I20260928 08:37:32.680481 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:32.661627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36885 (local address 127.0.97.1:50247)
0928 08:37:32.662407 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:32.662451 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:32.664150 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:32.664913 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:32.664922 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:32.665583 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:32.666194 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:32.666212 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:32.667467 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:32.667474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:32.668864 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:32.668875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:32.669684 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:32.669692 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:32.669929 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:32.670683 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:32.670705 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:32.672827 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:32.676626 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:32.676634 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:32.676648 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:32.677083 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:32.677474 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:32.677477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:32.677479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:32.677656 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:32.678299 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:32.678305 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:32.678697 (+ 392us) client_negotiation.cc:769] Sending connection context
0928 08:37:32.678934 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 08:37:32.679329 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":276,"threads_started":1}
I20260928 08:37:32.682129 4192 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36885
I20260928 08:37:32.682484 4192 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:32.683451 4192 heartbeater.cc:511] Master 127.0.97.62:36885 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:37:32.685825 4003 ts_manager.cc:194] Registered new tserver with Master: e99d81a30f2b4c91b6c18f6d68539b09 (127.0.97.1:38345)
I20260928 08:37:32.687391 4003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:37:32.714201 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36885
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:33.053090 4198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:33.053438 4198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:33.053613 4198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:33.061061 4198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:33.061250 4198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:33.061480 4198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:37:33.071808 4198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:33.073107 4198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:33.074381 4198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:33.087620 4204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:33.087599 4206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:33.087877 4203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:33.088681 4198 server_base.cc:1034] running on GCE node
I20260928 08:37:33.089411 4198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:33.090317 4198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:33.091566 4198 hybrid_clock.cc:648] HybridClock initialized: now 1790584653091517 us; error 55 us; skew 500 ppm
I20260928 08:37:33.091909 4198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:33 dist-test-slave-w6c0 krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:33.095549 4198 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:37:33.097354 4198 webserver.cc:460] Webserver started at http://127.0.97.2:39493/ using document root <none> and password file <none>
I20260928 08:37:33.098187 4198 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:33.098361 4198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:33.098744 4198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:33.101915 4198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "c0061dee2a024ca3bb1ffb4e06005a3a"
format_stamp: "Formatted at 2026-09-28 08:37:33 on dist-test-slave-w6c0"
server_key: "fe6eb8db608008470fdb7a26f4eb3e1f"
server_key_iv: "49379d1e6d5320e68d888cdd024e73c0"
server_key_version: "encryptionkey@0"
I20260928 08:37:33.102873 4198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c0061dee2a024ca3bb1ffb4e06005a3a"
format_stamp: "Formatted at 2026-09-28 08:37:33 on dist-test-slave-w6c0"
server_key: "fe6eb8db608008470fdb7a26f4eb3e1f"
server_key_iv: "49379d1e6d5320e68d888cdd024e73c0"
server_key_version: "encryptionkey@0"
I20260928 08:37:33.108136 4198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:37:33.111698 4213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:33.113056 4198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:37:33.113229 4198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "c0061dee2a024ca3bb1ffb4e06005a3a"
format_stamp: "Formatted at 2026-09-28 08:37:33 on dist-test-slave-w6c0"
server_key: "fe6eb8db608008470fdb7a26f4eb3e1f"
server_key_iv: "49379d1e6d5320e68d888cdd024e73c0"
server_key_version: "encryptionkey@0"
I20260928 08:37:33.113462 4198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:33.128860 4198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:33.133231 4198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:33.133531 4198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:33.134573 4198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:33.135949 4198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:33.136051 4198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:33.136214 4198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:33.136291 4198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:33.170351 4198 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38287
I20260928 08:37:33.170385 4325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38287 every 8 connection(s)
I20260928 08:37:33.171712 4198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:37:33.173619 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 4198
I20260928 08:37:33.173821 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:37:33.174288 388 external_mini_cluster.cc:1428] Setting key d44492f14aaa226d25f1500cdec11435
Sep 28 08:37:33 dist-test-slave-w6c0 krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:33.194031 4194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:33.174636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50621 (local address 127.0.97.62:36885)
0928 08:37:33.174856 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:33.174863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:33.176200 (+ 1337us) server_negotiation.cc:407] Connection header received
0928 08:37:33.178022 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:33.178031 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:33.178121 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:33.178228 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:33.179787 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:33.180934 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:33.182834 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:33.183285 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:33.183394 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:33.187721 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:33.187727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:33.187731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:33.187764 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:33.190787 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:33.191564 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:33.191571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:33.191574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:33.191762 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:33.192190 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:33.192196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:33.192202 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:33.192475 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:33.192630 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:33.193417 (+ 787us) server_negotiation.cc:299] Negotiation successful
0928 08:37:33.193647 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 08:37:33.195359 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:33.175195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36885 (local address 127.0.97.2:50621)
0928 08:37:33.176035 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:33.176086 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:33.177757 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:33.178449 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:33.178457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:33.179046 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:33.179618 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:33.179635 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:33.181118 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:33.181126 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:33.182685 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:33.182703 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:33.183540 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:33.183549 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:33.183914 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:33.184709 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:33.184745 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:33.187445 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:33.190936 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:33.190946 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:33.190965 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:33.191407 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:33.191865 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:33.191870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:33.191873 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:33.192074 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:33.192609 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:33.192617 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:33.193058 (+ 441us) client_negotiation.cc:769] Sending connection context
0928 08:37:33.193319 (+ 261us) client_negotiation.cc:240] Negotiation successful
0928 08:37:33.193911 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":366,"threads_started":1}
I20260928 08:37:33.202260 4326 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36885
I20260928 08:37:33.202798 4326 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:33.203792 4326 heartbeater.cc:511] Master 127.0.97.62:36885 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 08:37:33.205729 4003 ts_manager.cc:194] Registered new tserver with Master: c0061dee2a024ca3bb1ffb4e06005a3a (127.0.97.2:38287)
I20260928 08:37:33.206897 4003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:37:33.231137 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36885
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:33.565166 4331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:33.565490 4331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:33.565642 4331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:33.573192 4331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:33.573304 4331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:33.573484 4331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:37:33.583359 4331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:33.584705 4331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:33.586139 4331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:33.599992 4336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:33.599949 4339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:33.599911 4337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:33.600785 4331 server_base.cc:1034] running on GCE node
I20260928 08:37:33.601466 4331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:33.602555 4331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:33.603813 4331 hybrid_clock.cc:648] HybridClock initialized: now 1790584653603746 us; error 61 us; skew 500 ppm
I20260928 08:37:33.604122 4331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:33 dist-test-slave-w6c0 krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:33.607327 4331 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:37:33.608929 4331 webserver.cc:460] Webserver started at http://127.0.97.3:36599/ using document root <none> and password file <none>
I20260928 08:37:33.609613 4331 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:33.609714 4331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:33.609994 4331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:33.613093 4331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "26431583919d4cc1acb5290ff049086f"
format_stamp: "Formatted at 2026-09-28 08:37:33 on dist-test-slave-w6c0"
server_key: "fc0bb2de0b736d630d5d7743bc6e5bea"
server_key_iv: "8de465facc9aff7727cbd55378244c93"
server_key_version: "encryptionkey@0"
I20260928 08:37:33.613905 4331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "26431583919d4cc1acb5290ff049086f"
format_stamp: "Formatted at 2026-09-28 08:37:33 on dist-test-slave-w6c0"
server_key: "fc0bb2de0b736d630d5d7743bc6e5bea"
server_key_iv: "8de465facc9aff7727cbd55378244c93"
server_key_version: "encryptionkey@0"
I20260928 08:37:33.619552 4331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 08:37:33.623224 4346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:33.624540 4331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:37:33.624714 4331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "26431583919d4cc1acb5290ff049086f"
format_stamp: "Formatted at 2026-09-28 08:37:33 on dist-test-slave-w6c0"
server_key: "fc0bb2de0b736d630d5d7743bc6e5bea"
server_key_iv: "8de465facc9aff7727cbd55378244c93"
server_key_version: "encryptionkey@0"
I20260928 08:37:33.624930 4331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:33.665551 4331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:33.669790 4331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:33.670125 4331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:33.671221 4331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:33.672605 4331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:33.672719 4331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:33.672861 4331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:33.672935 4331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:33.690834 4192 heartbeater.cc:503] Master 127.0.97.62:36885 was elected leader, sending a full tablet report...
I20260928 08:37:33.706475 4331 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34537
I20260928 08:37:33.706568 4458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34537 every 8 connection(s)
I20260928 08:37:33.707799 4331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:37:33.711678 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 4331
I20260928 08:37:33.711875 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:37:33.712409 388 external_mini_cluster.cc:1428] Setting key d62198f42159474927775d69964471c0
Sep 28 08:37:33 dist-test-slave-w6c0 krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:33.728415 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:33.710396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33447 (local address 127.0.97.62:36885)
0928 08:37:33.710823 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:33.710827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:33.712056 (+ 1229us) server_negotiation.cc:407] Connection header received
0928 08:37:33.714321 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:33.714326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:33.714416 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:33.714567 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:33.716205 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:33.717085 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:33.718795 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:33.719208 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:33.719302 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:33.722665 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:33.722670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:33.722673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:33.722702 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:33.725150 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:33.725920 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:33.725923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:33.725926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:33.726087 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:33.726567 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:33.726571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:33.726573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:33.726862 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:33.727013 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:33.727844 (+ 831us) server_negotiation.cc:299] Negotiation successful
0928 08:37:33.728076 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":232,"threads_started":1}
I20260928 08:37:33.729488 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:33.711011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36885 (local address 127.0.97.3:33447)
0928 08:37:33.711851 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:33.711910 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:33.713992 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:33.714780 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:33.714789 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:33.715402 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:33.716052 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:33.716074 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:33.717206 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:33.717209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:33.718663 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:33.718675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:33.719391 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:33.719396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:33.719595 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:33.720224 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:33.720244 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:33.722386 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:33.725299 (+ 2913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:33.725306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:33.725318 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:33.725800 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:33.726200 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:33.726204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:33.726206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:33.726375 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:33.726994 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:33.727000 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:33.727443 (+ 443us) client_negotiation.cc:769] Sending connection context
0928 08:37:33.727683 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 08:37:33.728230 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":331,"threads_started":1}
I20260928 08:37:33.730831 4459 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36885
I20260928 08:37:33.731144 4459 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:33.731860 4459 heartbeater.cc:511] Master 127.0.97.62:36885 requested a full tablet report, sending...
I20260928 08:37:33.733237 4003 ts_manager.cc:194] Registered new tserver with Master: 26431583919d4cc1acb5290ff049086f (127.0.97.3:34537)
I20260928 08:37:33.734069 4003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33447
I20260928 08:37:33.740281 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:37:33 dist-test-slave-w6c0 krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:37:33 dist-test-slave-w6c0 krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:37:33.788937 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:33.773661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50022 (local address 127.0.97.1:38345)
0928 08:37:33.774300 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:33.774307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:33.774333 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:37:33.774393 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:33.774400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:33.774698 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:33.774830 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:33.776182 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:33.777180 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:33.778453 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:33.779136 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:33.779244 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:33.783162 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:33.783174 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:33.783190 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:33.783227 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:33.786132 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:33.786858 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:33.786865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:33.786867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:33.787001 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:33.787355 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:33.787358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:33.787360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:33.787720 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:33.787962 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:33.788255 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 08:37:33.788539 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":349,"threads_started":1}
Sep 28 08:37:33 dist-test-slave-w6c0 krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:33.807222 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:33.793537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.0.97.62:36885)
0928 08:37:33.793734 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:33.793740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:33.794062 (+ 322us) server_negotiation.cc:407] Connection header received
0928 08:37:33.794183 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:33.794187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:33.794278 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:33.794367 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:33.795562 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:33.796272 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:33.797478 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:33.797863 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:33.797965 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:33.802138 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:33.802145 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:33.802148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:33.802180 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:33.804820 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:33.805444 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:33.805448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:33.805450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:33.805556 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:33.805904 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:33.805908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:33.805910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:33.806234 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:33.806436 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:33.806819 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 08:37:33.806992 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 08:37:33.807778 4003 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49082
I20260928 08:37:33.808866 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 4064
I20260928 08:37:33.821686 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:33.809729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50034 (local address 127.0.97.1:38345)
0928 08:37:33.809939 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:33.809947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:33.810200 (+ 253us) server_negotiation.cc:407] Connection header received
0928 08:37:33.810376 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:33.810388 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:33.810451 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:33.810576 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:33.811785 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:33.812454 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:33.813651 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:33.814102 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:33.814235 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:33.816454 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:33.816459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:33.816462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:33.816492 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:33.819206 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:33.819835 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:33.819849 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:33.819851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:33.819966 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:33.820337 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:33.820343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:33.820347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:33.820591 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:33.820855 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:33.821154 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 08:37:33.821375 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:37:33.822256 4187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:33.906391 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 4064
I20260928 08:37:33.923241 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 4198
Sep 28 08:37:33 dist-test-slave-w6c0 krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:37:33.939098 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:33.924250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56978 (local address 127.0.97.2:38287)
0928 08:37:33.924811 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:33.924818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:33.924843 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:37:33.924901 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:33.924906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:33.925127 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:33.925269 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:33.926432 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:33.927491 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:33.928864 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:33.929517 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:33.929624 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:33.933742 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:33.933753 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:33.933769 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:33.933803 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:33.936541 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:33.937189 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:33.937195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:33.937198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:33.937329 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:33.937708 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:33.937711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:33.937713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:33.938032 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:33.938256 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:33.938613 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 08:37:33.938808 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":297,"threads_started":1}
I20260928 08:37:33.940395 4321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:34.025426 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 4198
I20260928 08:37:34.041855 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 4331
Sep 28 08:37:34 dist-test-slave-w6c0 krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:37:34.060093 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:34.043003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.0.97.3:34537)
0928 08:37:34.043641 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:34.043648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:34.043676 (+ 28us) server_negotiation.cc:407] Connection header received
0928 08:37:34.043736 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:34.043742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:34.043962 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:34.044144 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:34.045452 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:34.046485 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:34.048481 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:34.049096 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:34.049208 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:34.054039 (+ 4831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:34.054053 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:34.054069 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:34.054111 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:34.057257 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:34.057962 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:34.057970 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:34.057972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:34.058092 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:34.058457 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:34.058463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:34.058466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:34.058867 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:34.059205 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:34.059478 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 08:37:34.059753 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":334,"threads_started":1}
I20260928 08:37:34.061425 4454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:34.146867 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 4331
I20260928 08:37:34.164252 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 3971
I20260928 08:37:34.165531 4033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:34.269759 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 3971
2026-09-28T08:37:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3961 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:37:34 dist-test-slave-w6c0 krb5kdc[4479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:37:34 dist-test-slave-w6c0 krb5kdc[4479](info): set up 2 sockets
Sep 28 08:37:34 dist-test-slave-w6c0 krb5kdc[4479](info): commencing operation
krb5kdc: starting...
W20260928 08:37:35.354934 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:37:35 dist-test-slave-w6c0 krb5kdc[4479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:37:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:37:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:37:35.553316 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35199
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:35.865042 4492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:35.865345 4492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:35.865437 4492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:35.872543 4492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:37:35.872642 4492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:35.872689 4492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:35.872737 4492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:37:35.872778 4492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:37:35.882231 4492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:35.883463 4492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:35.884639 4492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:35.895701 4500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:35.895763 4497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:35.895790 4498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:35.896737 4492 server_base.cc:1034] running on GCE node
I20260928 08:37:35.897461 4492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:35.898862 4492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:35.900084 4492 hybrid_clock.cc:648] HybridClock initialized: now 1790584655900052 us; error 46 us; skew 500 ppm
I20260928 08:37:35.900427 4492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:35 dist-test-slave-w6c0 krb5kdc[4479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:35.903333 4492 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:37:35.904434 4492 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:35.904558 4492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:35.904862 4492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:35.907773 4492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "9025aee0fc43469d813644cdc8812e52"
format_stamp: "Formatted at 2026-09-28 08:37:35 on dist-test-slave-w6c0"
server_key: "e0bfbde15803f8e8e4df8c17850b8648"
server_key_iv: "1b3e934cf255776ac8fbd9e02ae8059b"
server_key_version: "encryptionkey@0"
I20260928 08:37:35.908543 4492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "9025aee0fc43469d813644cdc8812e52"
format_stamp: "Formatted at 2026-09-28 08:37:35 on dist-test-slave-w6c0"
server_key: "e0bfbde15803f8e8e4df8c17850b8648"
server_key_iv: "1b3e934cf255776ac8fbd9e02ae8059b"
server_key_version: "encryptionkey@0"
I20260928 08:37:35.913408 4492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:37:35.916658 4506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:35.917860 4492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 08:37:35.918032 4492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "9025aee0fc43469d813644cdc8812e52"
format_stamp: "Formatted at 2026-09-28 08:37:35 on dist-test-slave-w6c0"
server_key: "e0bfbde15803f8e8e4df8c17850b8648"
server_key_iv: "1b3e934cf255776ac8fbd9e02ae8059b"
server_key_version: "encryptionkey@0"
I20260928 08:37:35.918246 4492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:35.943406 4492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:35.947250 4492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:35.947504 4492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:35.964449 4492 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35199
I20260928 08:37:35.964470 4557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35199 every 8 connection(s)
I20260928 08:37:35.965318 4492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:37:35.968755 4558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:35.970656 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 4492
I20260928 08:37:35.970872 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:37:35.971450 388 external_mini_cluster.cc:1428] Setting key ca9597cb7229d2c2cef5a63daf21ac62
I20260928 08:37:35.981105 4558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52: Bootstrap starting.
I20260928 08:37:35.984440 4558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:35.985379 4558 log.cc:824] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52: Log is configured to *not* fsync() on all Append() calls
Sep 28 08:37:35 dist-test-slave-w6c0 krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:35.988037 4558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52: No bootstrap required, opened a new log
I20260928 08:37:35.994005 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:35.973410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56602 (local address 127.0.97.62:35199)
0928 08:37:35.974196 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:35.974212 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:35.974268 (+ 56us) server_negotiation.cc:407] Connection header received
0928 08:37:35.976534 (+ 2266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:35.976559 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:35.977023 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:35.977447 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:35.978832 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:35.980723 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:35.982062 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:35.983166 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:35.983332 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:35.987202 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:35.987209 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:35.987222 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:35.987253 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:35.990227 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:35.990955 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:35.990961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:35.990967 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:35.991122 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:35.991491 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:35.991500 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:35.991502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:35.991994 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:35.992323 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:35.992683 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 08:37:35.993070 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":345,"threads_started":1}
I20260928 08:37:35.995656 4558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9025aee0fc43469d813644cdc8812e52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } }
I20260928 08:37:35.995991 4558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:35.996109 4558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9025aee0fc43469d813644cdc8812e52, State: Initialized, Role: FOLLOWER
I20260928 08:37:35.996740 4558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9025aee0fc43469d813644cdc8812e52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } }
I20260928 08:37:35.996990 4558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:37:35.997116 4558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:37:35.997285 4558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:35.999440 4558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9025aee0fc43469d813644cdc8812e52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } }
I20260928 08:37:35.999919 4558 leader_election.cc:302] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9025aee0fc43469d813644cdc8812e52; no voters:
I20260928 08:37:36.000480 4558 leader_election.cc:288] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:37:36.000738 4563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:36.001435 4563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52 [term 1 LEADER]: Becoming Leader. State: Replica: 9025aee0fc43469d813644cdc8812e52, State: Running, Role: LEADER
I20260928 08:37:36.001916 4563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9025aee0fc43469d813644cdc8812e52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } }
I20260928 08:37:36.002485 4558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:37:36.004875 4564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9025aee0fc43469d813644cdc8812e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9025aee0fc43469d813644cdc8812e52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } }
I20260928 08:37:36.004976 4565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9025aee0fc43469d813644cdc8812e52. Latest consensus state: current_term: 1 leader_uuid: "9025aee0fc43469d813644cdc8812e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9025aee0fc43469d813644cdc8812e52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35199 } } }
I20260928 08:37:36.005270 4564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:36.005273 4565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:36.008224 4571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:37:36.012348 4571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:37:36.020480 4571 catalog_manager.cc:1347] Generated new cluster ID: 51d91595907a4e30beed617c8a77f314
I20260928 08:37:36.020592 4571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:37:36.036584 4571 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:37:36.037919 4571 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:37:36.047349 4571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52: Generated new TSK 0
I20260928 08:37:36.048139 4571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:37:36.093752 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35199
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:36.416033 4584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:36.416316 4584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:36.416505 4584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:36.424043 4584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:36.424180 4584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:36.424367 4584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:37:36.434167 4584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:36.435494 4584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:36.436817 4584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:36.449194 4589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:36.449182 4590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:36.449374 4592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:36.450027 4584 server_base.cc:1034] running on GCE node
I20260928 08:37:36.450726 4584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:36.451583 4584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:36.452778 4584 hybrid_clock.cc:648] HybridClock initialized: now 1790584656452720 us; error 51 us; skew 500 ppm
I20260928 08:37:36.453086 4584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:36 dist-test-slave-w6c0 krb5kdc[4479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:36.456359 4584 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:37:36.457556 4584 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:36.457672 4584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:36.458007 4584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:36.460958 4584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "8f17811741b3453494eab6e82e245d12"
format_stamp: "Formatted at 2026-09-28 08:37:36 on dist-test-slave-w6c0"
server_key: "2ed131b0084ef86d09068653f5696953"
server_key_iv: "f8138056362677c9f547194a0fd27b4a"
server_key_version: "encryptionkey@0"
I20260928 08:37:36.461795 4584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8f17811741b3453494eab6e82e245d12"
format_stamp: "Formatted at 2026-09-28 08:37:36 on dist-test-slave-w6c0"
server_key: "2ed131b0084ef86d09068653f5696953"
server_key_iv: "f8138056362677c9f547194a0fd27b4a"
server_key_version: "encryptionkey@0"
I20260928 08:37:36.466986 4584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 08:37:36.470600 4598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:36.471974 4584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:37:36.472141 4584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "8f17811741b3453494eab6e82e245d12"
format_stamp: "Formatted at 2026-09-28 08:37:36 on dist-test-slave-w6c0"
server_key: "2ed131b0084ef86d09068653f5696953"
server_key_iv: "f8138056362677c9f547194a0fd27b4a"
server_key_version: "encryptionkey@0"
I20260928 08:37:36.472369 4584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:36.490103 4584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:36.494470 4584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:36.494803 4584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:36.495756 4584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:36.497087 4584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:36.497212 4584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:36.497339 4584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:36.497430 4584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:36.530603 4584 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41247
I20260928 08:37:36.530689 4710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41247 every 8 connection(s)
I20260928 08:37:36.531472 4584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:37:36.533661 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 4584
I20260928 08:37:36.533878 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:37:36.534689 388 external_mini_cluster.cc:1428] Setting key 04fb1b9a2264d247232cac79df434379
Sep 28 08:37:36 dist-test-slave-w6c0 krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:36.552047 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:36.533971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58077 (local address 127.0.97.62:35199)
0928 08:37:36.534463 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:36.534467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:36.535653 (+ 1186us) server_negotiation.cc:407] Connection header received
0928 08:37:36.537894 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:36.537900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:36.537986 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:36.538098 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:36.539662 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:36.540383 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:36.541907 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:36.542303 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:36.542425 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:36.546354 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:36.546359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:36.546363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:36.546395 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:36.548996 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:36.549766 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:36.549775 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:36.549778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:36.549891 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:36.550289 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:36.550294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:36.550298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:36.550654 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:36.550803 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:36.551468 (+ 665us) server_negotiation.cc:299] Negotiation successful
0928 08:37:36.551688 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":283,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:37:36.553038 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:36.534686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.1:58077)
0928 08:37:36.535487 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:36.535548 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:36.537615 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:36.538327 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:36.538335 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:36.538925 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:36.539501 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:36.539517 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:36.540535 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:36.540540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:36.541764 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:36.541776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:36.542550 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:36.542560 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:36.542784 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:36.543545 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:36.543567 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:36.546081 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:36.549141 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:36.549150 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:36.549167 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:36.549635 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:36.549994 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:36.550000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:36.550004 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:36.550178 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:36.550778 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:36.550785 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:36.551125 (+ 340us) client_negotiation.cc:769] Sending connection context
0928 08:37:36.551382 (+ 257us) client_negotiation.cc:240] Negotiation successful
0928 08:37:36.551912 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":318,"threads_started":1}
I20260928 08:37:36.554811 4711 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35199
I20260928 08:37:36.555110 4711 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 08:37:36.555881 4711 heartbeater.cc:511] Master 127.0.97.62:35199 requested a full tablet report, sending...
I20260928 08:37:36.557943 4523 ts_manager.cc:194] Registered new tserver with Master: 8f17811741b3453494eab6e82e245d12 (127.0.97.1:41247)
I20260928 08:37:36.559414 4523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:37:36.578613 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35199
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:36.899287 4717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:36.899565 4717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:36.899659 4717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:36.906967 4717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:36.907220 4717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:36.907402 4717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:37:36.917359 4717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:36.918620 4717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:36.919837 4717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:36.934773 4725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:36.934827 4722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:36.934801 4723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:36.935650 4717 server_base.cc:1034] running on GCE node
I20260928 08:37:36.936434 4717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:36.937439 4717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:36.938660 4717 hybrid_clock.cc:648] HybridClock initialized: now 1790584656938614 us; error 55 us; skew 500 ppm
I20260928 08:37:36.938973 4717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:36 dist-test-slave-w6c0 krb5kdc[4479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:36.941911 4717 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:37:36.943012 4717 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:36.943099 4717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:36.943336 4717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:36.946110 4717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "0fd5d957ede9403887788e490edbb9c1"
format_stamp: "Formatted at 2026-09-28 08:37:36 on dist-test-slave-w6c0"
server_key: "1bb2175b9504c71da9395feef77eda5b"
server_key_iv: "43e13a3ffb4f538c497663d12e15facb"
server_key_version: "encryptionkey@0"
I20260928 08:37:36.946935 4717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0fd5d957ede9403887788e490edbb9c1"
format_stamp: "Formatted at 2026-09-28 08:37:36 on dist-test-slave-w6c0"
server_key: "1bb2175b9504c71da9395feef77eda5b"
server_key_iv: "43e13a3ffb4f538c497663d12e15facb"
server_key_version: "encryptionkey@0"
I20260928 08:37:36.951824 4717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 08:37:36.955103 4731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:36.956403 4717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:37:36.956533 4717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "0fd5d957ede9403887788e490edbb9c1"
format_stamp: "Formatted at 2026-09-28 08:37:36 on dist-test-slave-w6c0"
server_key: "1bb2175b9504c71da9395feef77eda5b"
server_key_iv: "43e13a3ffb4f538c497663d12e15facb"
server_key_version: "encryptionkey@0"
I20260928 08:37:36.956703 4717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:36.991094 4717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:36.995098 4717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:36.995318 4717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:36.996208 4717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:36.997455 4717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:36.997546 4717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:36.997632 4717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:36.997686 4717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:37.030807 4717 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44791
I20260928 08:37:37.030867 4843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44791 every 8 connection(s)
I20260928 08:37:37.031687 4717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:37:37.037415 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 4717
I20260928 08:37:37.037636 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:37:37.038051 388 external_mini_cluster.cc:1428] Setting key 31983d71bf2eed37831375c4dd54f071
Sep 28 08:37:37 dist-test-slave-w6c0 krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:37.052944 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:37.034162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53931 (local address 127.0.97.62:35199)
0928 08:37:37.034377 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:37.034383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:37.035616 (+ 1233us) server_negotiation.cc:407] Connection header received
0928 08:37:37.037392 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:37.037398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:37.037492 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:37.037605 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:37.039368 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.040277 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.041810 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.042191 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.042319 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:37.045746 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:37.045751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:37.045754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:37.045786 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:37.049435 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:37.050245 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:37.050254 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:37.050260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:37.050424 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:37.050937 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:37.050945 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:37.050951 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:37.051345 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:37.051493 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:37.052217 (+ 724us) server_negotiation.cc:299] Negotiation successful
0928 08:37:37.052476 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:37:37.053845 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:37.034725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.2:53931)
0928 08:37:37.035435 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:37.035485 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:37.037114 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:37.037868 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:37.037876 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:37.038444 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:37.039206 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:37.039224 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.040428 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.040432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:37.041666 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:37.041676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.042404 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.042411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:37.042684 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:37.043379 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:37.043401 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:37.045532 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:37.049590 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:37.049598 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:37.049613 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:37.050107 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:37.050625 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:37.050629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:37.050632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:37.050811 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:37.051472 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:37.051478 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:37.051824 (+ 346us) client_negotiation.cc:769] Sending connection context
0928 08:37:37.052064 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 08:37:37.052495 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":300,"threads_started":1}
I20260928 08:37:37.055648 4844 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35199
I20260928 08:37:37.055954 4844 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:37.056605 4844 heartbeater.cc:511] Master 127.0.97.62:35199 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 08:37:37.058061 4523 ts_manager.cc:194] Registered new tserver with Master: 0fd5d957ede9403887788e490edbb9c1 (127.0.97.2:44791)
I20260928 08:37:37.058905 4523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:37:37.085444 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35199
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:37.413435 4849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:37.413698 4849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:37.413791 4849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:37.420792 4849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:37.420895 4849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:37.421015 4849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:37:37.431277 4849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:37.432556 4849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:37.433786 4849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:37.446584 4855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:37.446622 4857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:37.446882 4854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:37.448385 4849 server_base.cc:1034] running on GCE node
I20260928 08:37:37.448959 4849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:37.449929 4849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:37.451187 4849 hybrid_clock.cc:648] HybridClock initialized: now 1790584657451130 us; error 41 us; skew 500 ppm
I20260928 08:37:37.451478 4849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:37 dist-test-slave-w6c0 krb5kdc[4479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:37.454475 4849 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:37:37.455641 4849 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:37.455729 4849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:37.455986 4849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:37.458768 4849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "e9bcb3be77984494a2e9b20703227ce0"
format_stamp: "Formatted at 2026-09-28 08:37:37 on dist-test-slave-w6c0"
server_key: "6607cb2d59400176ca64c25fee618a6c"
server_key_iv: "7fee565785de00c6a6a9ac74daa318be"
server_key_version: "encryptionkey@0"
I20260928 08:37:37.459555 4849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e9bcb3be77984494a2e9b20703227ce0"
format_stamp: "Formatted at 2026-09-28 08:37:37 on dist-test-slave-w6c0"
server_key: "6607cb2d59400176ca64c25fee618a6c"
server_key_iv: "7fee565785de00c6a6a9ac74daa318be"
server_key_version: "encryptionkey@0"
I20260928 08:37:37.464542 4849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:37:37.467916 4863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:37.469323 4849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 08:37:37.469470 4849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "e9bcb3be77984494a2e9b20703227ce0"
format_stamp: "Formatted at 2026-09-28 08:37:37 on dist-test-slave-w6c0"
server_key: "6607cb2d59400176ca64c25fee618a6c"
server_key_iv: "7fee565785de00c6a6a9ac74daa318be"
server_key_version: "encryptionkey@0"
I20260928 08:37:37.469648 4849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:37.504665 4849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:37.509004 4849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:37.509232 4849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:37.510113 4849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:37.511456 4849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:37.511581 4849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:37.511709 4849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:37.511792 4849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:37.545147 4849 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42085
I20260928 08:37:37.545178 4975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42085 every 8 connection(s)
I20260928 08:37:37.545969 4849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:37:37.554723 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 4849
I20260928 08:37:37.554917 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:37:37.555356 388 external_mini_cluster.cc:1428] Setting key 4c2de107736a2b5ce04ee875c44ba046
Sep 28 08:37:37 dist-test-slave-w6c0 krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:37.562202 4711 heartbeater.cc:503] Master 127.0.97.62:35199 was elected leader, sending a full tablet report...
I20260928 08:37:37.566002 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:37.548384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49297 (local address 127.0.97.62:35199)
0928 08:37:37.548600 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:37.548607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:37.549661 (+ 1054us) server_negotiation.cc:407] Connection header received
0928 08:37:37.551439 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:37.551443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:37.551515 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:37.551602 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:37.552930 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.553697 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.555368 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.555965 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.556093 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:37.560257 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:37.560263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:37.560266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:37.560298 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:37.563175 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:37.563948 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:37.563952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:37.563955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:37.564076 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:37.564471 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:37.564475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:37.564478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:37.564744 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:37.564861 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:37.565609 (+ 748us) server_negotiation.cc:299] Negotiation successful
0928 08:37:37.565767 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:37:37.567026 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:37.548842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35199 (local address 127.0.97.3:49297)
0928 08:37:37.549512 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:37.549550 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:37.551200 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:37.551780 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:37.551787 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:37.552277 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:37.552790 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:37.552805 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.553815 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.553819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:37.555230 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:37.555241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.556183 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.556188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:37.556415 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:37.557393 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:37.557415 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:37.560045 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:37.563299 (+ 3254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:37.563306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:37.563318 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:37.563830 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:37.564174 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:37.564177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:37.564179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:37.564335 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:37.564889 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:37.564896 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:37.565228 (+ 332us) client_negotiation.cc:769] Sending connection context
0928 08:37:37.565499 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 08:37:37.565855 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":292,"threads_started":1}
I20260928 08:37:37.568420 4976 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35199
I20260928 08:37:37.568729 4976 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:37.569481 4976 heartbeater.cc:511] Master 127.0.97.62:35199 requested a full tablet report, sending...
I20260928 08:37:37.570998 4523 ts_manager.cc:194] Registered new tserver with Master: e9bcb3be77984494a2e9b20703227ce0 (127.0.97.3:42085)
I20260928 08:37:37.571776 4523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49297
I20260928 08:37:37.582702 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:37:37.598735 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:37.587107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56604 (local address 127.0.97.62:35199)
0928 08:37:37.587259 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:37.587263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:37.587773 (+ 510us) server_negotiation.cc:407] Connection header received
0928 08:37:37.587899 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:37.587903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:37.587958 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:37.588037 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:37.589373 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.590055 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.591306 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.591715 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.591811 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:37.594047 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:37.594051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:37.594054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:37.594081 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:37.596393 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:37.597011 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:37.597015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:37.597017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:37.597124 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:37.597404 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:37.597408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:37.597410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:37.597723 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:37.597894 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:37.598264 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 08:37:37.598423 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 08:37:37.602689 4523 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:37:37.609992 4523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:37:37.647166 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:37.622488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57230 (local address 127.0.97.1:41247)
0928 08:37:37.633675 (+ 11187us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:37.633689 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:37.637396 (+ 3707us) server_negotiation.cc:407] Connection header received
0928 08:37:37.637525 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:37.637536 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:37.637861 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:37.638109 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:37.638947 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.641537 (+ 2590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.644051 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.645357 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.645517 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:37.645765 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:37.646620 (+ 855us) server_negotiation.cc:299] Negotiation successful
0928 08:37:37.646807 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10927,"thread_start_us":10755,"threads_started":1}
I20260928 08:37:37.647274 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:37.622000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44791 (local address 127.0.0.1:59246)
0928 08:37:37.634218 (+ 12218us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:37.634272 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:37.634492 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:37.640719 (+ 6227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:37.640724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:37.640773 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:37.641211 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:37.641222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.642617 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.642622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:37.644420 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:37.644435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.646441 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.646447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:37.646630 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:37.646659 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 08:37:37.646737 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 08:37:37.646848 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11863,"mutex_wait_us":16,"spinlock_wait_cycles":6528,"thread_start_us":15002,"threads_started":1}
I20260928 08:37:37.647778 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:37.622195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41247 (local address 127.0.0.1:57230)
0928 08:37:37.637263 (+ 15068us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:37.637277 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:37.637370 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:37.638149 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:37.638160 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:37.638201 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:37.638708 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:37.638725 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.641664 (+ 2939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.641672 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:37.643892 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:37.643903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.645889 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.645895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:37.646112 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:37.646166 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 08:37:37.646291 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 08:37:37.646492 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14976,"thread_start_us":11489,"threads_started":1}
I20260928 08:37:37.647989 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:37.622202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59246 (local address 127.0.97.2:44791)
0928 08:37:37.640185 (+ 17983us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:37.640196 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:37.640224 (+ 28us) server_negotiation.cc:407] Connection header received
0928 08:37:37.640310 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:37.640317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:37.640525 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:37.641057 (+ 532us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:37.641384 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.642439 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.644584 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.646006 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.646122 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:37.646302 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:37.647488 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0928 08:37:37.647670 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17824,"thread_start_us":17676,"threads_started":1}
I20260928 08:37:37.648344 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:37.625128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.0.97.3:42085)
0928 08:37:37.639061 (+ 13933us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:37.639075 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:37.639639 (+ 564us) server_negotiation.cc:407] Connection header received
0928 08:37:37.639835 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:37.639846 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:37.640066 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:37.640295 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:37.641758 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.643339 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.645335 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.646685 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.646810 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:37.647025 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:37.647878 (+ 853us) server_negotiation.cc:299] Negotiation successful
0928 08:37:37.648008 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13728,"thread_start_us":13593,"threads_started":1}
I20260928 08:37:37.648712 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:37.624405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42085 (local address 127.0.0.1:43160)
0928 08:37:37.639513 (+ 15108us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:37.639528 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:37.639666 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:37.640639 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:37.640646 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:37.640675 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:37.641569 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:37.641582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.643486 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.643491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:37.645186 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:37.645197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.647170 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.647175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:37.647344 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:37.647389 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 08:37:37.647472 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 08:37:37.647599 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14998,"mutex_wait_us":7,"thread_start_us":14864,"threads_started":1}
I20260928 08:37:37.651516 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 8a91690064c649b7ab5b3afd9f4bf07e (DEFAULT_TABLE table=test-table [id=724a6d8cee5946cd9e128d51b7d7ca4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:37.651643 4779 tablet_service.cc:1467] Processing CreateTablet for tablet 8a91690064c649b7ab5b3afd9f4bf07e (DEFAULT_TABLE table=test-table [id=724a6d8cee5946cd9e128d51b7d7ca4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:37.651913 4911 tablet_service.cc:1467] Processing CreateTablet for tablet 8a91690064c649b7ab5b3afd9f4bf07e (DEFAULT_TABLE table=test-table [id=724a6d8cee5946cd9e128d51b7d7ca4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:37.652794 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a91690064c649b7ab5b3afd9f4bf07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:37.652943 4779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a91690064c649b7ab5b3afd9f4bf07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:37.653010 4911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a91690064c649b7ab5b3afd9f4bf07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:37.663906 4994 tablet_bootstrap.cc:492] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12: Bootstrap starting.
I20260928 08:37:37.667105 4995 tablet_bootstrap.cc:492] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0: Bootstrap starting.
I20260928 08:37:37.667204 4994 tablet_bootstrap.cc:654] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:37.668514 4994 log.cc:824] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:37.668797 4996 tablet_bootstrap.cc:492] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1: Bootstrap starting.
I20260928 08:37:37.670774 4995 tablet_bootstrap.cc:654] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:37.671876 4996 tablet_bootstrap.cc:654] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:37.671993 4995 log.cc:824] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:37.672217 4994 tablet_bootstrap.cc:492] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12: No bootstrap required, opened a new log
I20260928 08:37:37.672842 4994 ts_tablet_manager.cc:1397] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260928 08:37:37.673125 4996 log.cc:824] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:37.677373 4995 tablet_bootstrap.cc:492] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0: No bootstrap required, opened a new log
I20260928 08:37:37.677636 4995 ts_tablet_manager.cc:1397] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260928 08:37:37.677779 4996 tablet_bootstrap.cc:492] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1: No bootstrap required, opened a new log
I20260928 08:37:37.678013 4996 ts_tablet_manager.cc:1397] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260928 08:37:37.684011 4994 raft_consensus.cc:348] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd5d957ede9403887788e490edbb9c1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "8f17811741b3453494eab6e82e245d12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41247 } } peers { permanent_uuid: "e9bcb3be77984494a2e9b20703227ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42085 } }
I20260928 08:37:37.684459 4994 raft_consensus.cc:374] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:37.684629 4994 raft_consensus.cc:729] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f17811741b3453494eab6e82e245d12, State: Initialized, Role: FOLLOWER
I20260928 08:37:37.684692 4996 raft_consensus.cc:348] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd5d957ede9403887788e490edbb9c1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "8f17811741b3453494eab6e82e245d12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41247 } } peers { permanent_uuid: "e9bcb3be77984494a2e9b20703227ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42085 } }
I20260928 08:37:37.685065 4996 raft_consensus.cc:374] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:37.685159 4996 raft_consensus.cc:729] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fd5d957ede9403887788e490edbb9c1, State: Initialized, Role: FOLLOWER
I20260928 08:37:37.685375 4994 consensus_queue.cc:260] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd5d957ede9403887788e490edbb9c1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "8f17811741b3453494eab6e82e245d12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41247 } } peers { permanent_uuid: "e9bcb3be77984494a2e9b20703227ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42085 } }
I20260928 08:37:37.685730 4996 consensus_queue.cc:260] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd5d957ede9403887788e490edbb9c1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "8f17811741b3453494eab6e82e245d12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41247 } } peers { permanent_uuid: "e9bcb3be77984494a2e9b20703227ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42085 } }
I20260928 08:37:37.686877 4844 heartbeater.cc:503] Master 127.0.97.62:35199 was elected leader, sending a full tablet report...
I20260928 08:37:37.687575 4996 ts_tablet_manager.cc:1428] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1: Time spent starting tablet: real 0.009s user 0.009s sys 0.001s
I20260928 08:37:37.688259 4995 raft_consensus.cc:348] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd5d957ede9403887788e490edbb9c1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "8f17811741b3453494eab6e82e245d12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41247 } } peers { permanent_uuid: "e9bcb3be77984494a2e9b20703227ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42085 } }
I20260928 08:37:37.688782 4995 raft_consensus.cc:374] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:37.688905 4995 raft_consensus.cc:729] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9bcb3be77984494a2e9b20703227ce0, State: Initialized, Role: FOLLOWER
I20260928 08:37:37.689610 4995 consensus_queue.cc:260] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd5d957ede9403887788e490edbb9c1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "8f17811741b3453494eab6e82e245d12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41247 } } peers { permanent_uuid: "e9bcb3be77984494a2e9b20703227ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42085 } }
I20260928 08:37:37.690948 4976 heartbeater.cc:503] Master 127.0.97.62:35199 was elected leader, sending a full tablet report...
I20260928 08:37:37.691512 4995 ts_tablet_manager.cc:1428] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260928 08:37:37.693269 4994 ts_tablet_manager.cc:1428] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12: Time spent starting tablet: real 0.020s user 0.006s sys 0.010s
I20260928 08:37:37.742931 5002 raft_consensus.cc:482] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:37:37.743268 5002 raft_consensus.cc:504] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd5d957ede9403887788e490edbb9c1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "8f17811741b3453494eab6e82e245d12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41247 } } peers { permanent_uuid: "e9bcb3be77984494a2e9b20703227ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42085 } }
I20260928 08:37:37.745026 5002 leader_election.cc:288] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fd5d957ede9403887788e490edbb9c1 (127.0.97.2:44791), 8f17811741b3453494eab6e82e245d12 (127.0.97.1:41247)
I20260928 08:37:37.750802 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:37.745374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36375 (local address 127.0.97.2:44791)
0928 08:37:37.745565 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:37.745573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:37.745833 (+ 260us) server_negotiation.cc:407] Connection header received
0928 08:37:37.745911 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:37.745916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:37.745989 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:37.746071 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:37.746699 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.747573 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.749167 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.749904 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.750048 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:37.750154 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:37.750403 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 08:37:37.750550 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:37:37.750806 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:37.745242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44791 (local address 127.0.97.3:36375)
0928 08:37:37.745544 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:37.745565 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:37.745710 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:37.746250 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:37.746254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:37.746283 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:37.746525 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:37.746538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.747730 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.747736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:37.749015 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:37.749026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.750069 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.750074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:37.750254 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:37.750277 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 08:37:37.750378 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 08:37:37.750508 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260928 08:37:37.751739 4799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a91690064c649b7ab5b3afd9f4bf07e" candidate_uuid: "e9bcb3be77984494a2e9b20703227ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fd5d957ede9403887788e490edbb9c1" is_pre_election: true
I20260928 08:37:37.752017 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:37.745852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47121 (local address 127.0.97.1:41247)
0928 08:37:37.746100 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:37.746109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:37.746781 (+ 672us) server_negotiation.cc:407] Connection header received
0928 08:37:37.746984 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:37.746988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:37.747046 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:37.747144 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:37.747828 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.748573 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.750212 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.751149 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.751298 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:37.751397 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:37.751651 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 08:37:37.751780 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 08:37:37.752070 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:37.745410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41247 (local address 127.0.97.3:47121)
0928 08:37:37.746655 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:37.746669 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:37.746823 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:37.747166 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:37.747170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:37.747193 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:37.747497 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:37.747505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.748731 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.748737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:37.750048 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:37.750059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.751336 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.751341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:37.751496 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:37.751518 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 08:37:37.751620 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 08:37:37.751740 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1121,"thread_start_us":915,"threads_started":1}
I20260928 08:37:37.752290 4799 raft_consensus.cc:2454] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9bcb3be77984494a2e9b20703227ce0 in term 0.
I20260928 08:37:37.752944 4865 leader_election.cc:302] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fd5d957ede9403887788e490edbb9c1, e9bcb3be77984494a2e9b20703227ce0; no voters:
I20260928 08:37:37.753046 4666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a91690064c649b7ab5b3afd9f4bf07e" candidate_uuid: "e9bcb3be77984494a2e9b20703227ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f17811741b3453494eab6e82e245d12" is_pre_election: true
I20260928 08:37:37.753309 5002 raft_consensus.cc:2781] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:37:37.753468 5002 raft_consensus.cc:482] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:37:37.753557 5002 raft_consensus.cc:3037] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:37.753608 4666 raft_consensus.cc:2454] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9bcb3be77984494a2e9b20703227ce0 in term 0.
I20260928 08:37:37.755869 5002 raft_consensus.cc:504] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd5d957ede9403887788e490edbb9c1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "8f17811741b3453494eab6e82e245d12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41247 } } peers { permanent_uuid: "e9bcb3be77984494a2e9b20703227ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42085 } }
I20260928 08:37:37.756520 5002 leader_election.cc:288] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [CANDIDATE]: Term 1 election: Requested vote from peers 0fd5d957ede9403887788e490edbb9c1 (127.0.97.2:44791), 8f17811741b3453494eab6e82e245d12 (127.0.97.1:41247)
I20260928 08:37:37.756916 4799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a91690064c649b7ab5b3afd9f4bf07e" candidate_uuid: "e9bcb3be77984494a2e9b20703227ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fd5d957ede9403887788e490edbb9c1"
I20260928 08:37:37.757130 4799 raft_consensus.cc:3037] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:37.757156 4666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a91690064c649b7ab5b3afd9f4bf07e" candidate_uuid: "e9bcb3be77984494a2e9b20703227ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f17811741b3453494eab6e82e245d12"
I20260928 08:37:37.757362 4666 raft_consensus.cc:3037] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:37.759521 4666 raft_consensus.cc:2454] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9bcb3be77984494a2e9b20703227ce0 in term 1.
I20260928 08:37:37.759505 4799 raft_consensus.cc:2454] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9bcb3be77984494a2e9b20703227ce0 in term 1.
I20260928 08:37:37.759946 4865 leader_election.cc:302] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fd5d957ede9403887788e490edbb9c1, e9bcb3be77984494a2e9b20703227ce0; no voters:
I20260928 08:37:37.760258 5002 raft_consensus.cc:2781] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:37.760859 5002 raft_consensus.cc:686] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [term 1 LEADER]: Becoming Leader. State: Replica: e9bcb3be77984494a2e9b20703227ce0, State: Running, Role: LEADER
I20260928 08:37:37.761353 5002 consensus_queue.cc:237] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd5d957ede9403887788e490edbb9c1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } } peers { permanent_uuid: "8f17811741b3453494eab6e82e245d12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41247 } } peers { permanent_uuid: "e9bcb3be77984494a2e9b20703227ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42085 } }
I20260928 08:37:37.766078 4522 catalog_manager.cc:5495] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9bcb3be77984494a2e9b20703227ce0 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e9bcb3be77984494a2e9b20703227ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fd5d957ede9403887788e490edbb9c1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f17811741b3453494eab6e82e245d12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9bcb3be77984494a2e9b20703227ce0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42085 } health_report { overall_health: HEALTHY } } }
W20260928 08:37:37.783710 4845 tablet.cc:2369] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:37:37.783905 4712 tablet.cc:2369] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:37:37.788522 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:37.783005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.0.97.3:42085)
0928 08:37:37.783261 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:37.783268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:37.783437 (+ 169us) server_negotiation.cc:407] Connection header received
0928 08:37:37.783795 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:37.783802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:37.783972 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:37.784129 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:37:37.784636 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.785352 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.786734 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.787121 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.787213 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:37.787625 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:37:37.788103 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:37:37.788207 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:37.788275 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 08:37:37.788337 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 08:37:37.796490 4799 raft_consensus.cc:1261] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1 [term 1 FOLLOWER]: Refusing update from remote peer e9bcb3be77984494a2e9b20703227ce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:37.796494 4666 raft_consensus.cc:1261] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12 [term 1 FOLLOWER]: Refusing update from remote peer e9bcb3be77984494a2e9b20703227ce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260928 08:37:37.797389 4977 tablet.cc:2369] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:37:37.797586 5004 consensus_queue.cc:1035] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fd5d957ede9403887788e490edbb9c1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:37.798027 5002 consensus_queue.cc:1035] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f17811741b3453494eab6e82e245d12" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:37.823606 4522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:37.823858 4522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:37.827221 4522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9025aee0fc43469d813644cdc8812e52: Sending DeleteTablet for 3 replicas of tablet 8a91690064c649b7ab5b3afd9f4bf07e
I20260928 08:37:37.828297 4779 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a91690064c649b7ab5b3afd9f4bf07e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59246
I20260928 08:37:37.828377 4911 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a91690064c649b7ab5b3afd9f4bf07e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43160
I20260928 08:37:37.828377 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a91690064c649b7ab5b3afd9f4bf07e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57230
I20260928 08:37:37.830384 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 4584
I20260928 08:37:37.839634 5018 tablet_replica.cc:330] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12: stopping tablet replica
I20260928 08:37:37.840368 5018 raft_consensus.cc:2229] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:37.841081 5018 raft_consensus.cc:2258] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:37.841720 5017 tablet_replica.cc:330] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1: stopping tablet replica
I20260928 08:37:37.842152 5017 raft_consensus.cc:2229] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:37.842660 5017 raft_consensus.cc:2258] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:37.845060 5018 ts_tablet_manager.cc:1905] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:37.845062 5017 ts_tablet_manager.cc:1905] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:37.852190 5018 ts_tablet_manager.cc:1918] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:37:37.852469 5018 log.cc:1196] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-0/wal/wals/8a91690064c649b7ab5b3afd9f4bf07e
I20260928 08:37:37.853085 5018 ts_tablet_manager.cc:1939] T 8a91690064c649b7ab5b3afd9f4bf07e P 8f17811741b3453494eab6e82e245d12: Deleting consensus metadata
I20260928 08:37:37.853996 5017 ts_tablet_manager.cc:1918] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:37:37.854255 5017 log.cc:1196] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/8a91690064c649b7ab5b3afd9f4bf07e
I20260928 08:37:37.854903 5017 ts_tablet_manager.cc:1939] T 8a91690064c649b7ab5b3afd9f4bf07e P 0fd5d957ede9403887788e490edbb9c1: Deleting consensus metadata
I20260928 08:37:37.855222 4507 catalog_manager.cc:4841] TS 8f17811741b3453494eab6e82e245d12 (127.0.97.1:41247): tablet 8a91690064c649b7ab5b3afd9f4bf07e (table test-table [id=724a6d8cee5946cd9e128d51b7d7ca4f]) successfully deleted
I20260928 08:37:37.856822 4508 catalog_manager.cc:4841] TS 0fd5d957ede9403887788e490edbb9c1 (127.0.97.2:44791): tablet 8a91690064c649b7ab5b3afd9f4bf07e (table test-table [id=724a6d8cee5946cd9e128d51b7d7ca4f]) successfully deleted
I20260928 08:37:37.857257 5019 tablet_replica.cc:330] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0: stopping tablet replica
I20260928 08:37:37.857762 5019 raft_consensus.cc:2229] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:37:37.858430 5019 raft_consensus.cc:2258] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:37.860675 5019 ts_tablet_manager.cc:1905] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 08:37:37 dist-test-slave-w6c0 krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:37:37.868026 5019 ts_tablet_manager.cc:1918] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:37:37.868191 5019 log.cc:1196] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/8a91690064c649b7ab5b3afd9f4bf07e
I20260928 08:37:37.868760 5019 ts_tablet_manager.cc:1939] T 8a91690064c649b7ab5b3afd9f4bf07e P e9bcb3be77984494a2e9b20703227ce0: Deleting consensus metadata
I20260928 08:37:37.870680 4510 catalog_manager.cc:4841] TS e9bcb3be77984494a2e9b20703227ce0 (127.0.97.3:42085): tablet 8a91690064c649b7ab5b3afd9f4bf07e (table test-table [id=724a6d8cee5946cd9e128d51b7d7ca4f]) successfully deleted
I20260928 08:37:37.871538 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:37.834449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57242 (local address 127.0.97.1:41247)
0928 08:37:37.834657 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:37.834663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:37.857034 (+ 22371us) server_negotiation.cc:407] Connection header received
0928 08:37:37.857383 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:37.857388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:37.857467 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:37.857574 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:37.858956 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.859777 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.861155 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.861642 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.861773 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:37.865527 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:37.865539 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:37.865553 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:37.865592 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:37.868529 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:37.869285 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:37.869294 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:37.869296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:37.869456 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:37.869968 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:37.869974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:37.869977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:37.870423 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:37.870779 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:37.871059 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 08:37:37.871223 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 08:37:37.872273 4706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:37.961310 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 4584
I20260928 08:37:37.984750 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 4717
Sep 28 08:37:37 dist-test-slave-w6c0 krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:37:38.003016 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:37.985540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59258 (local address 127.0.97.2:44791)
0928 08:37:37.985742 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:37.985749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:37.985889 (+ 140us) server_negotiation.cc:407] Connection header received
0928 08:37:37.986077 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:37.986083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:37.986164 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:37.986277 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:37.988102 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.989242 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.991097 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:37.991711 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:37.991833 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:37.996957 (+ 5124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:37.996972 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:37.996994 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:37.997036 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:38.000199 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:38.000898 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:38.000905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:38.000908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:38.001036 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:38.001419 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:38.001424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:38.001432 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:38.001787 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:38.002102 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:38.002370 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 08:37:38.002651 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:37:38.003763 4839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:38.092180 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 4717
I20260928 08:37:38.114848 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 4849
Sep 28 08:37:38 dist-test-slave-w6c0 krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:37:38.132745 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:38.115662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43180 (local address 127.0.97.3:42085)
0928 08:37:38.115883 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:38.115889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:38.115914 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:37:38.115987 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:38.115992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:38.116073 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:38.116190 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:38.117470 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:38.118884 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:38.120178 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:38.120867 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:38.121019 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:38.125468 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:38.125485 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:38.125507 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:38.125549 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:38.129669 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:38.130363 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:38.130372 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:38.130375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:38.130607 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:38.130953 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:38.130959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:38.130962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:38.131425 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:38.131762 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:38.131922 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 08:37:38.132308 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 08:37:38.133569 4971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:38.257144 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 4849
I20260928 08:37:38.276369 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 4492
I20260928 08:37:38.277485 4553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:38.385571 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 4492
2026-09-28T08:37:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4115 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T08:37:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:37:38Z Disabled control of system clock
I20260928 08:37:38.453076 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38779
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 08:37:38.779014 5028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:38.779621 5028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:38.779801 5028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:38.788362 5028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:37:38.788512 5028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:38.788553 5028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:38.788609 5028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:37:38.788655 5028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:37:38.798575 5028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38779
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:38.799907 5028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:38.801709 5028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:38.813776 5033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:38.813772 5034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:38.814026 5036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:38.814658 5028 server_base.cc:1034] running on GCE node
I20260928 08:37:38.815462 5028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:38.816974 5028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:38.818179 5028 hybrid_clock.cc:648] HybridClock initialized: now 1790584658818142 us; error 36 us; skew 500 ppm
I20260928 08:37:38.818598 5028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:37:38.820899 5028 webserver.cc:460] Webserver started at http://127.0.97.62:40171/ using document root <none> and password file <none>
I20260928 08:37:38.821624 5028 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:38.821733 5028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:38.822062 5028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:38.825069 5028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "eddde00e8f414b8fadd8a58f39a3bb22"
format_stamp: "Formatted at 2026-09-28 08:37:38 on dist-test-slave-w6c0"
server_key: "c784263b0417acf3fce7028b30cda1aa"
server_key_iv: "4908b76b7c687a452f4a10aa1da3bab0"
server_key_version: "encryptionkey@0"
I20260928 08:37:38.825866 5028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "eddde00e8f414b8fadd8a58f39a3bb22"
format_stamp: "Formatted at 2026-09-28 08:37:38 on dist-test-slave-w6c0"
server_key: "c784263b0417acf3fce7028b30cda1aa"
server_key_iv: "4908b76b7c687a452f4a10aa1da3bab0"
server_key_version: "encryptionkey@0"
I20260928 08:37:38.831251 5028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:37:38.835117 5042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:38.836462 5028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:37:38.836645 5028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "eddde00e8f414b8fadd8a58f39a3bb22"
format_stamp: "Formatted at 2026-09-28 08:37:38 on dist-test-slave-w6c0"
server_key: "c784263b0417acf3fce7028b30cda1aa"
server_key_iv: "4908b76b7c687a452f4a10aa1da3bab0"
server_key_version: "encryptionkey@0"
I20260928 08:37:38.836879 5028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:38.842918 5028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:38.843735 5028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:38.844053 5028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:38.862090 5028 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38779
I20260928 08:37:38.862129 5093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38779 every 8 connection(s)
I20260928 08:37:38.863474 5028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:37:38.867318 5094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:38.873157 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 5028
I20260928 08:37:38.873377 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:37:38.874002 388 external_mini_cluster.cc:1428] Setting key edae0c112e3d86d9d6cd28a11ae78b80
I20260928 08:37:38.879006 5094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22: Bootstrap starting.
I20260928 08:37:38.883077 5094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:38.883805 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:38.876223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60826 (local address 127.0.97.62:38779)
0928 08:37:38.877284 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:38.877298 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:38.877351 (+ 53us) server_negotiation.cc:407] Connection header received
0928 08:37:38.879691 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:38.879720 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:38.880245 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:38.880712 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:38.881766 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:38.881774 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:38.881790 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:37:38.881806 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:38.881903 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:37:38.881924 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:38.882138 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:38.882265 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 08:37:38.882558 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":308,"threads_started":1}
I20260928 08:37:38.884379 5094 log.cc:824] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:38.887323 5094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22: No bootstrap required, opened a new log
I20260928 08:37:38.894695 5094 raft_consensus.cc:348] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddde00e8f414b8fadd8a58f39a3bb22" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38779 } }
I20260928 08:37:38.895074 5094 raft_consensus.cc:374] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:38.895193 5094 raft_consensus.cc:729] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eddde00e8f414b8fadd8a58f39a3bb22, State: Initialized, Role: FOLLOWER
I20260928 08:37:38.895788 5094 consensus_queue.cc:260] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddde00e8f414b8fadd8a58f39a3bb22" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38779 } }
I20260928 08:37:38.896025 5094 raft_consensus.cc:388] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:37:38.896158 5094 raft_consensus.cc:482] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:37:38.896369 5094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:38.898612 5094 raft_consensus.cc:504] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddde00e8f414b8fadd8a58f39a3bb22" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38779 } }
I20260928 08:37:38.899120 5094 leader_election.cc:302] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eddde00e8f414b8fadd8a58f39a3bb22; no voters:
I20260928 08:37:38.899796 5094 leader_election.cc:288] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:37:38.899928 5099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:38.900676 5099 raft_consensus.cc:686] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22 [term 1 LEADER]: Becoming Leader. State: Replica: eddde00e8f414b8fadd8a58f39a3bb22, State: Running, Role: LEADER
I20260928 08:37:38.901248 5099 consensus_queue.cc:237] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddde00e8f414b8fadd8a58f39a3bb22" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38779 } }
I20260928 08:37:38.901969 5094 sys_catalog.cc:564] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:37:38.904163 5100 sys_catalog.cc:455] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eddde00e8f414b8fadd8a58f39a3bb22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddde00e8f414b8fadd8a58f39a3bb22" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38779 } } }
I20260928 08:37:38.904621 5100 sys_catalog.cc:458] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:38.904626 5101 sys_catalog.cc:455] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eddde00e8f414b8fadd8a58f39a3bb22. Latest consensus state: current_term: 1 leader_uuid: "eddde00e8f414b8fadd8a58f39a3bb22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eddde00e8f414b8fadd8a58f39a3bb22" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38779 } } }
I20260928 08:37:38.905133 5101 sys_catalog.cc:458] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:38.906807 5104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:37:38.911710 5104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:37:38.920784 5104 catalog_manager.cc:1347] Generated new cluster ID: 72d1936506e9466ba6f9eafa11e7165f
I20260928 08:37:38.920914 5104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:37:38.940923 5104 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:37:38.941459 5104 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:37:38.950188 5104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22: Generated new TSK 0
I20260928 08:37:38.951126 5104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:37:38.956887 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38779
--builtin_ntp_servers=127.0.97.20:45593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 08:37:39.312989 5118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:39.313264 5118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:39.313426 5118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:39.320578 5118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:39.320670 5118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:39.320847 5118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:37:39.329695 5118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:39.331158 5118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:39.332921 5118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:39.346145 5124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:39.346179 5126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:39.346344 5123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:39.347563 5118 server_base.cc:1034] running on GCE node
I20260928 08:37:39.348097 5118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:39.348976 5118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:39.350174 5118 hybrid_clock.cc:648] HybridClock initialized: now 1790584659350134 us; error 35 us; skew 500 ppm
I20260928 08:37:39.350544 5118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:37:39.352658 5118 webserver.cc:460] Webserver started at http://127.0.97.1:34087/ using document root <none> and password file <none>
I20260928 08:37:39.353328 5118 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:39.353453 5118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:39.353776 5118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:39.356612 5118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "f786416b6fc04883b4b9ec5b1ebedb9c"
format_stamp: "Formatted at 2026-09-28 08:37:39 on dist-test-slave-w6c0"
server_key: "d29721b768aabad9853779e819814972"
server_key_iv: "213d81092a7e4a1d1e3ca36d8922a8e8"
server_key_version: "encryptionkey@0"
I20260928 08:37:39.357395 5118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f786416b6fc04883b4b9ec5b1ebedb9c"
format_stamp: "Formatted at 2026-09-28 08:37:39 on dist-test-slave-w6c0"
server_key: "d29721b768aabad9853779e819814972"
server_key_iv: "213d81092a7e4a1d1e3ca36d8922a8e8"
server_key_version: "encryptionkey@0"
I20260928 08:37:39.362264 5118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 08:37:39.365655 5132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:39.366961 5118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:37:39.367121 5118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "f786416b6fc04883b4b9ec5b1ebedb9c"
format_stamp: "Formatted at 2026-09-28 08:37:39 on dist-test-slave-w6c0"
server_key: "d29721b768aabad9853779e819814972"
server_key_iv: "213d81092a7e4a1d1e3ca36d8922a8e8"
server_key_version: "encryptionkey@0"
I20260928 08:37:39.367337 5118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:39.373037 5118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:39.373799 5118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:39.374065 5118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:39.375023 5118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:39.376365 5118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:39.376497 5118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:39.376612 5118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:39.376693 5118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:39.409058 5118 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33409
I20260928 08:37:39.409107 5244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33409 every 8 connection(s)
I20260928 08:37:39.410310 5118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:37:39.411038 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 5118
I20260928 08:37:39.411258 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:37:39.412001 388 external_mini_cluster.cc:1428] Setting key f8bd0b9d428090f3af1d53c233ab6358
I20260928 08:37:39.415015 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38779
--builtin_ntp_servers=127.0.97.20:45593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 08:37:39.418807 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:39.413028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53085 (local address 127.0.97.62:38779)
0928 08:37:39.413464 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:39.413470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:39.414862 (+ 1392us) server_negotiation.cc:407] Connection header received
0928 08:37:39.416557 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:39.416562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:39.416625 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:39.416746 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:39.418026 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:39.418031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:39.418036 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:37:39.418057 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:39.418095 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:37:39.418106 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:39.418223 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:39.418417 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 08:37:39.418484 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":216,"threads_started":1}
I20260928 08:37:39.419831 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:39.413708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38779 (local address 127.0.97.1:53085)
0928 08:37:39.414725 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:39.414762 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:39.416311 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:39.416972 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:39.416980 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:39.417065 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:39.417744 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:37:39.417746 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:39.417781 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:37:39.417857 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:39.418195 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:39.418204 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:39.418211 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 08:37:39.418473 (+ 262us) client_negotiation.cc:240] Negotiation successful
0928 08:37:39.418726 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":341,"threads_started":1}
I20260928 08:37:39.421201 5245 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38779
I20260928 08:37:39.421483 5245 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:39.422111 5245 heartbeater.cc:511] Master 127.0.97.62:38779 requested a full tablet report, sending...
I20260928 08:37:39.424118 5059 ts_manager.cc:194] Registered new tserver with Master: f786416b6fc04883b4b9ec5b1ebedb9c (127.0.97.1:33409)
W20260928 08:37:39.730793 5249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:39.731082 5249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:39.731240 5249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:39.738394 5249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:39.738487 5249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:39.738682 5249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:37:39.747759 5249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:39.749022 5249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:39.750746 5249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:39.764367 5255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:39.764408 5257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:39.764330 5254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:39.765327 5249 server_base.cc:1034] running on GCE node
I20260928 08:37:39.766028 5249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:39.767076 5249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:39.768311 5249 hybrid_clock.cc:648] HybridClock initialized: now 1790584659768259 us; error 59 us; skew 500 ppm
I20260928 08:37:39.768642 5249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:37:39.771239 5249 webserver.cc:460] Webserver started at http://127.0.97.2:42187/ using document root <none> and password file <none>
I20260928 08:37:39.771986 5249 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:39.772109 5249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:39.772416 5249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:39.775417 5249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "b8dc18fbf1954e76a2281d23c744b9e5"
format_stamp: "Formatted at 2026-09-28 08:37:39 on dist-test-slave-w6c0"
server_key: "43b549305684309c33d7c8ad83fc6d14"
server_key_iv: "df1287ce211d45e9535df92e26226cd8"
server_key_version: "encryptionkey@0"
I20260928 08:37:39.776196 5249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b8dc18fbf1954e76a2281d23c744b9e5"
format_stamp: "Formatted at 2026-09-28 08:37:39 on dist-test-slave-w6c0"
server_key: "43b549305684309c33d7c8ad83fc6d14"
server_key_iv: "df1287ce211d45e9535df92e26226cd8"
server_key_version: "encryptionkey@0"
I20260928 08:37:39.781065 5249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:37:39.784409 5263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:39.785696 5249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:37:39.785862 5249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "b8dc18fbf1954e76a2281d23c744b9e5"
format_stamp: "Formatted at 2026-09-28 08:37:39 on dist-test-slave-w6c0"
server_key: "43b549305684309c33d7c8ad83fc6d14"
server_key_iv: "df1287ce211d45e9535df92e26226cd8"
server_key_version: "encryptionkey@0"
I20260928 08:37:39.786079 5249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:39.791424 5249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:39.792189 5249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:39.792456 5249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:39.793320 5249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:39.794694 5249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:39.794806 5249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:39.794930 5249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:39.795027 5249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:39.826364 5249 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40391
I20260928 08:37:39.826398 5375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40391 every 8 connection(s)
I20260928 08:37:39.827730 5249 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:37:39.834452 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 5249
I20260928 08:37:39.834673 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:37:39.835251 388 external_mini_cluster.cc:1428] Setting key 699f631a7cae1ab619fde287a9d6473e
I20260928 08:37:39.836055 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:39.830276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42837 (local address 127.0.97.62:38779)
0928 08:37:39.830489 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:39.830495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:39.831994 (+ 1499us) server_negotiation.cc:407] Connection header received
0928 08:37:39.833574 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:39.833579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:39.833631 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:39.833724 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:39.835271 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:39.835276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:39.835279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:37:39.835295 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:39.835325 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:37:39.835337 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:39.835479 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:39.835678 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 08:37:39.835728 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:37:39.837188 5378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:39.830887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38779 (local address 127.0.97.2:42837)
0928 08:37:39.831788 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:39.831831 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:39.833321 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:39.833920 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:39.833928 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:39.834047 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:39.834922 (+ 875us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:37:39.834926 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:39.834977 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:37:39.835088 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:39.835454 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:39.835464 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:39.835474 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 08:37:39.835743 (+ 269us) client_negotiation.cc:240] Negotiation successful
0928 08:37:39.836026 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":273,"threads_started":1}
I20260928 08:37:39.838188 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38779
--builtin_ntp_servers=127.0.97.20:45593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 08:37:39.838551 5376 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38779
I20260928 08:37:39.838848 5376 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:39.839404 5376 heartbeater.cc:511] Master 127.0.97.62:38779 requested a full tablet report, sending...
I20260928 08:37:39.840804 5059 ts_manager.cc:194] Registered new tserver with Master: b8dc18fbf1954e76a2281d23c744b9e5 (127.0.97.2:40391)
W20260928 08:37:40.157744 5379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:40.158013 5379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:40.158102 5379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:40.165263 5379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:40.165364 5379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:40.165472 5379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:37:40.174539 5379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:40.176143 5379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:40.178565 5379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:40.190991 5385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:40.191015 5384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:40.192256 5387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:40.193113 5379 server_base.cc:1034] running on GCE node
I20260928 08:37:40.193679 5379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:40.194645 5379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:40.195833 5379 hybrid_clock.cc:648] HybridClock initialized: now 1790584660195777 us; error 48 us; skew 500 ppm
I20260928 08:37:40.196138 5379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:37:40.198300 5379 webserver.cc:460] Webserver started at http://127.0.97.3:35633/ using document root <none> and password file <none>
I20260928 08:37:40.199008 5379 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:40.199115 5379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:40.199437 5379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:40.202306 5379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "f71527a946064a3f8235ac6c970c4a5b"
format_stamp: "Formatted at 2026-09-28 08:37:40 on dist-test-slave-w6c0"
server_key: "637ffd90f79f682d208606a88ed0ebef"
server_key_iv: "d47bc6642e681f4e17ccf92c27254ba1"
server_key_version: "encryptionkey@0"
I20260928 08:37:40.203178 5379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f71527a946064a3f8235ac6c970c4a5b"
format_stamp: "Formatted at 2026-09-28 08:37:40 on dist-test-slave-w6c0"
server_key: "637ffd90f79f682d208606a88ed0ebef"
server_key_iv: "d47bc6642e681f4e17ccf92c27254ba1"
server_key_version: "encryptionkey@0"
I20260928 08:37:40.208441 5379 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:37:40.211694 5393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:40.212970 5379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:37:40.213146 5379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "f71527a946064a3f8235ac6c970c4a5b"
format_stamp: "Formatted at 2026-09-28 08:37:40 on dist-test-slave-w6c0"
server_key: "637ffd90f79f682d208606a88ed0ebef"
server_key_iv: "d47bc6642e681f4e17ccf92c27254ba1"
server_key_version: "encryptionkey@0"
I20260928 08:37:40.213370 5379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:40.218686 5379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:40.219440 5379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:40.219695 5379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:40.220562 5379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:40.221870 5379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:40.221982 5379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:40.222103 5379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:40.222190 5379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:40.253635 5379 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41265
I20260928 08:37:40.253683 5505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41265 every 8 connection(s)
I20260928 08:37:40.254973 5379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:37:40.256685 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 5379
I20260928 08:37:40.256846 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:37:40.257321 388 external_mini_cluster.cc:1428] Setting key 4955d7baddb542070aac2c82a4fac1c5
I20260928 08:37:40.263592 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.257519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32945 (local address 127.0.97.62:38779)
0928 08:37:40.257737 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:40.257743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:40.259879 (+ 2136us) server_negotiation.cc:407] Connection header received
0928 08:37:40.261588 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:40.261592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:40.261641 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:40.261758 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:40.263024 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:40.263029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:40.263032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:37:40.263047 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:40.263079 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:37:40.263091 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:40.263196 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:40.263345 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 08:37:40.263391 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:37:40.264488 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.258074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38779 (local address 127.0.97.3:32945)
0928 08:37:40.259694 (+ 1620us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:40.259749 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:40.261351 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:40.261939 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:40.261947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:40.262035 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:40.262727 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:37:40.262730 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:40.262765 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:37:40.262844 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:40.263181 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:40.263188 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:40.263193 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:37:40.263392 (+ 199us) client_negotiation.cc:240] Negotiation successful
0928 08:37:40.263575 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"thread_start_us":815,"threads_started":1}
I20260928 08:37:40.265672 5506 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38779
I20260928 08:37:40.265961 5506 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:40.266567 5506 heartbeater.cc:511] Master 127.0.97.62:38779 requested a full tablet report, sending...
I20260928 08:37:40.267814 5059 ts_manager.cc:194] Registered new tserver with Master: f71527a946064a3f8235ac6c970c4a5b (127.0.97.3:41265)
I20260928 08:37:40.272358 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:37:40.279072 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.276693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60828 (local address 127.0.97.62:38779)
0928 08:37:40.276875 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:40.276879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:40.277419 (+ 540us) server_negotiation.cc:407] Connection header received
0928 08:37:40.277542 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:40.277545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:40.277594 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:40.277678 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:40.278483 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:40.278487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:40.278490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:37:40.278540 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:40.278574 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:37:40.278592 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:40.278683 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:40.278841 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 08:37:40.278886 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 08:37:40.282553 5059 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:37:40.289191 5059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:37:40.305228 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.300301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40391 (local address 127.0.0.1:33674)
0928 08:37:40.301751 (+ 1450us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:40.301808 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:40.301982 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:40.303053 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:40.303063 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:40.303157 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:40.303944 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:37:40.303946 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:40.303970 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:37:40.304034 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:40.304443 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:40.304455 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:40.304466 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 08:37:40.304661 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 08:37:40.304762 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1162,"thread_start_us":1052,"threads_started":1}
I20260928 08:37:40.305527 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.300507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.0.97.2:40391)
0928 08:37:40.301173 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:40.301184 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:40.301916 (+ 732us) server_negotiation.cc:407] Connection header received
0928 08:37:40.302130 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:40.302139 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:40.302366 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:40.302540 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:40.304187 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:40.304197 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:40.304215 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:37:40.304234 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:40.304298 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:37:40.304327 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:40.304506 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:40.304634 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 08:37:40.304754 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":356,"threads_started":1}
I20260928 08:37:40.307324 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.301527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41098 (local address 127.0.97.1:33409)
0928 08:37:40.303023 (+ 1496us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:40.303034 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:40.303067 (+ 33us) server_negotiation.cc:407] Connection header received
0928 08:37:40.303150 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:40.303160 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:40.303786 (+ 626us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:40.305102 (+ 1316us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:40.306461 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:40.306466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:40.306475 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:37:40.306492 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:40.306588 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:37:40.306610 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:40.306736 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:40.306837 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 08:37:40.306924 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1296,"spinlock_wait_cycles":18688,"thread_start_us":1263,"threads_started":1}
I20260928 08:37:40.307375 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.301011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33409 (local address 127.0.0.1:41098)
0928 08:37:40.301751 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:40.301808 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:40.301955 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:40.304429 (+ 2474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:40.304437 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:40.304471 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:40.305258 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:37:40.305264 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:40.305286 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:37:40.305355 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:40.306701 (+ 1346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:40.306706 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:40.306710 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:37:40.306860 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 08:37:40.306910 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":691,"threads_started":1}
I20260928 08:37:40.309523 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.302136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41265 (local address 127.0.0.1:55858)
0928 08:37:40.302714 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:40.302729 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:40.302850 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:40.306974 (+ 4124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:40.306979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:40.307004 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:40.307679 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:37:40.307682 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:40.307697 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:37:40.307750 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:40.308979 (+ 1229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:40.308984 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:40.308989 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:37:40.309098 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 08:37:40.309145 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":300,"threads_started":1}
I20260928 08:37:40.310457 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.302541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.0.97.3:41265)
0928 08:37:40.306442 (+ 3901us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:40.306451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:40.306482 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:37:40.306577 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:40.306584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:40.306784 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:40.307421 (+ 637us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:40.308750 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:40.308757 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:40.308768 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:37:40.308788 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:40.308837 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:37:40.308854 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:40.309936 (+ 1082us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:40.309998 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 08:37:40.310082 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3749,"thread_start_us":3597,"threads_started":1}
I20260928 08:37:40.310124 5311 tablet_service.cc:1467] Processing CreateTablet for tablet e3bf013b67dd4191b0c8bc595e6fcd07 (DEFAULT_TABLE table=test-table [id=fc6e1ee3272540d08b5b16c159b07b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:40.310941 5180 tablet_service.cc:1467] Processing CreateTablet for tablet e3bf013b67dd4191b0c8bc595e6fcd07 (DEFAULT_TABLE table=test-table [id=fc6e1ee3272540d08b5b16c159b07b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:40.311657 5311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3bf013b67dd4191b0c8bc595e6fcd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:40.312150 5180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3bf013b67dd4191b0c8bc595e6fcd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:40.313791 5441 tablet_service.cc:1467] Processing CreateTablet for tablet e3bf013b67dd4191b0c8bc595e6fcd07 (DEFAULT_TABLE table=test-table [id=fc6e1ee3272540d08b5b16c159b07b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:40.315205 5441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3bf013b67dd4191b0c8bc595e6fcd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:40.322899 5524 tablet_bootstrap.cc:492] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5: Bootstrap starting.
I20260928 08:37:40.326843 5524 tablet_bootstrap.cc:654] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:40.326970 5525 tablet_bootstrap.cc:492] T e3bf013b67dd4191b0c8bc595e6fcd07 P f786416b6fc04883b4b9ec5b1ebedb9c: Bootstrap starting.
I20260928 08:37:40.328245 5524 log.cc:824] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:40.329721 5526 tablet_bootstrap.cc:492] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b: Bootstrap starting.
I20260928 08:37:40.330668 5525 tablet_bootstrap.cc:654] T e3bf013b67dd4191b0c8bc595e6fcd07 P f786416b6fc04883b4b9ec5b1ebedb9c: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:40.331383 5524 tablet_bootstrap.cc:492] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5: No bootstrap required, opened a new log
I20260928 08:37:40.331611 5524 ts_tablet_manager.cc:1397] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 08:37:40.331959 5525 log.cc:824] T e3bf013b67dd4191b0c8bc595e6fcd07 P f786416b6fc04883b4b9ec5b1ebedb9c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:40.333118 5526 tablet_bootstrap.cc:654] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:40.334288 5526 log.cc:824] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:40.334945 5525 tablet_bootstrap.cc:492] T e3bf013b67dd4191b0c8bc595e6fcd07 P f786416b6fc04883b4b9ec5b1ebedb9c: No bootstrap required, opened a new log
I20260928 08:37:40.335189 5525 ts_tablet_manager.cc:1397] T e3bf013b67dd4191b0c8bc595e6fcd07 P f786416b6fc04883b4b9ec5b1ebedb9c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260928 08:37:40.337129 5526 tablet_bootstrap.cc:492] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b: No bootstrap required, opened a new log
I20260928 08:37:40.337380 5526 ts_tablet_manager.cc:1397] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260928 08:37:40.339772 5524 raft_consensus.cc:348] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dc18fbf1954e76a2281d23c744b9e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "f71527a946064a3f8235ac6c970c4a5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41265 } } peers { permanent_uuid: "f786416b6fc04883b4b9ec5b1ebedb9c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33409 } }
I20260928 08:37:40.340227 5524 raft_consensus.cc:374] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:40.340322 5524 raft_consensus.cc:729] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8dc18fbf1954e76a2281d23c744b9e5, State: Initialized, Role: FOLLOWER
I20260928 08:37:40.340947 5524 consensus_queue.cc:260] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dc18fbf1954e76a2281d23c744b9e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "f71527a946064a3f8235ac6c970c4a5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41265 } } peers { permanent_uuid: "f786416b6fc04883b4b9ec5b1ebedb9c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33409 } }
I20260928 08:37:40.342082 5376 heartbeater.cc:503] Master 127.0.97.62:38779 was elected leader, sending a full tablet report...
I20260928 08:37:40.342749 5524 ts_tablet_manager.cc:1428] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260928 08:37:40.345062 5525 raft_consensus.cc:348] T e3bf013b67dd4191b0c8bc595e6fcd07 P f786416b6fc04883b4b9ec5b1ebedb9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dc18fbf1954e76a2281d23c744b9e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "f71527a946064a3f8235ac6c970c4a5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41265 } } peers { permanent_uuid: "f786416b6fc04883b4b9ec5b1ebedb9c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33409 } }
I20260928 08:37:40.345431 5525 raft_consensus.cc:374] T e3bf013b67dd4191b0c8bc595e6fcd07 P f786416b6fc04883b4b9ec5b1ebedb9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:40.345530 5525 raft_consensus.cc:729] T e3bf013b67dd4191b0c8bc595e6fcd07 P f786416b6fc04883b4b9ec5b1ebedb9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f786416b6fc04883b4b9ec5b1ebedb9c, State: Initialized, Role: FOLLOWER
I20260928 08:37:40.346132 5525 consensus_queue.cc:260] T e3bf013b67dd4191b0c8bc595e6fcd07 P f786416b6fc04883b4b9ec5b1ebedb9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dc18fbf1954e76a2281d23c744b9e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "f71527a946064a3f8235ac6c970c4a5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41265 } } peers { permanent_uuid: "f786416b6fc04883b4b9ec5b1ebedb9c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33409 } }
I20260928 08:37:40.347229 5245 heartbeater.cc:503] Master 127.0.97.62:38779 was elected leader, sending a full tablet report...
I20260928 08:37:40.347925 5525 ts_tablet_manager.cc:1428] T e3bf013b67dd4191b0c8bc595e6fcd07 P f786416b6fc04883b4b9ec5b1ebedb9c: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260928 08:37:40.348032 5526 raft_consensus.cc:348] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dc18fbf1954e76a2281d23c744b9e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "f71527a946064a3f8235ac6c970c4a5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41265 } } peers { permanent_uuid: "f786416b6fc04883b4b9ec5b1ebedb9c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33409 } }
I20260928 08:37:40.348459 5526 raft_consensus.cc:374] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:40.348580 5526 raft_consensus.cc:729] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71527a946064a3f8235ac6c970c4a5b, State: Initialized, Role: FOLLOWER
I20260928 08:37:40.349118 5526 consensus_queue.cc:260] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dc18fbf1954e76a2281d23c744b9e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "f71527a946064a3f8235ac6c970c4a5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41265 } } peers { permanent_uuid: "f786416b6fc04883b4b9ec5b1ebedb9c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33409 } }
I20260928 08:37:40.350095 5506 heartbeater.cc:503] Master 127.0.97.62:38779 was elected leader, sending a full tablet report...
I20260928 08:37:40.351270 5526 ts_tablet_manager.cc:1428] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260928 08:37:40.358124 5530 raft_consensus.cc:482] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:37:40.358378 5530 raft_consensus.cc:504] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dc18fbf1954e76a2281d23c744b9e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "f71527a946064a3f8235ac6c970c4a5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41265 } } peers { permanent_uuid: "f786416b6fc04883b4b9ec5b1ebedb9c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33409 } }
I20260928 08:37:40.359902 5530 leader_election.cc:288] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f71527a946064a3f8235ac6c970c4a5b (127.0.97.3:41265), f786416b6fc04883b4b9ec5b1ebedb9c (127.0.97.1:33409)
I20260928 08:37:40.363363 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.360321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51833 (local address 127.0.97.3:41265)
0928 08:37:40.360473 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:40.360479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:40.361163 (+ 684us) server_negotiation.cc:407] Connection header received
0928 08:37:40.361288 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:40.361294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:40.361348 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:40.361446 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:40.362669 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:40.362674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:40.362679 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:37:40.362699 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:40.362739 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:37:40.362757 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:40.362901 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:40.363054 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 08:37:40.363123 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 08:37:40.363456 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.360254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35887 (local address 127.0.97.1:33409)
0928 08:37:40.360421 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:40.360427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:40.361125 (+ 698us) server_negotiation.cc:407] Connection header received
0928 08:37:40.361288 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:40.361294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:40.361348 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:40.361446 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:40.362715 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:40.362719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:40.362722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:37:40.362737 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:40.362785 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:37:40.362800 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:40.362935 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:40.363149 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 08:37:40.363204 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:37:40.363519 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.360191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41265 (local address 127.0.97.2:51833)
0928 08:37:40.361012 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:40.361026 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:40.361161 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:40.361481 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:40.361486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:40.361510 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:40.362430 (+ 920us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:37:40.362435 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:40.362455 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:37:40.362505 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:40.362870 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:40.362875 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:40.362881 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 08:37:40.363033 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 08:37:40.363083 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"spinlock_wait_cycles":31104,"thread_start_us":447,"threads_started":1}
I20260928 08:37:40.364177 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.360137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33409 (local address 127.0.97.2:35887)
0928 08:37:40.361004 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:40.361025 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:40.361161 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:40.361481 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:40.361486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:40.361510 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:40.362465 (+ 955us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:37:40.362468 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:40.362479 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:37:40.362541 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:40.362990 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:40.363012 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:40.363019 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 08:37:40.363821 (+ 802us) client_negotiation.cc:240] Negotiation successful
0928 08:37:40.363873 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"spinlock_wait_cycles":1280,"thread_start_us":485,"threads_started":1}
I20260928 08:37:40.364432 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3bf013b67dd4191b0c8bc595e6fcd07" candidate_uuid: "b8dc18fbf1954e76a2281d23c744b9e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71527a946064a3f8235ac6c970c4a5b" is_pre_election: true
I20260928 08:37:40.364880 5461 raft_consensus.cc:2454] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8dc18fbf1954e76a2281d23c744b9e5 in term 0.
I20260928 08:37:40.365056 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3bf013b67dd4191b0c8bc595e6fcd07" candidate_uuid: "b8dc18fbf1954e76a2281d23c744b9e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f786416b6fc04883b4b9ec5b1ebedb9c" is_pre_election: true
I20260928 08:37:40.365459 5267 leader_election.cc:302] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8dc18fbf1954e76a2281d23c744b9e5, f71527a946064a3f8235ac6c970c4a5b; no voters:
I20260928 08:37:40.365494 5200 raft_consensus.cc:2454] T e3bf013b67dd4191b0c8bc595e6fcd07 P f786416b6fc04883b4b9ec5b1ebedb9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8dc18fbf1954e76a2281d23c744b9e5 in term 0.
I20260928 08:37:40.365784 5530 raft_consensus.cc:2781] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:37:40.365933 5530 raft_consensus.cc:482] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:37:40.366048 5530 raft_consensus.cc:3037] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:40.368319 5530 raft_consensus.cc:504] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dc18fbf1954e76a2281d23c744b9e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "f71527a946064a3f8235ac6c970c4a5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41265 } } peers { permanent_uuid: "f786416b6fc04883b4b9ec5b1ebedb9c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33409 } }
I20260928 08:37:40.368849 5530 leader_election.cc:288] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [CANDIDATE]: Term 1 election: Requested vote from peers f71527a946064a3f8235ac6c970c4a5b (127.0.97.3:41265), f786416b6fc04883b4b9ec5b1ebedb9c (127.0.97.1:33409)
I20260928 08:37:40.369158 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3bf013b67dd4191b0c8bc595e6fcd07" candidate_uuid: "b8dc18fbf1954e76a2281d23c744b9e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71527a946064a3f8235ac6c970c4a5b"
I20260928 08:37:40.369339 5461 raft_consensus.cc:3037] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:40.369313 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3bf013b67dd4191b0c8bc595e6fcd07" candidate_uuid: "b8dc18fbf1954e76a2281d23c744b9e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f786416b6fc04883b4b9ec5b1ebedb9c"
I20260928 08:37:40.369499 5200 raft_consensus.cc:3037] T e3bf013b67dd4191b0c8bc595e6fcd07 P f786416b6fc04883b4b9ec5b1ebedb9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:40.371942 5461 raft_consensus.cc:2454] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8dc18fbf1954e76a2281d23c744b9e5 in term 1.
I20260928 08:37:40.371954 5200 raft_consensus.cc:2454] T e3bf013b67dd4191b0c8bc595e6fcd07 P f786416b6fc04883b4b9ec5b1ebedb9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8dc18fbf1954e76a2281d23c744b9e5 in term 1.
I20260928 08:37:40.372321 5266 leader_election.cc:302] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8dc18fbf1954e76a2281d23c744b9e5, f786416b6fc04883b4b9ec5b1ebedb9c; no voters:
I20260928 08:37:40.372637 5530 raft_consensus.cc:2781] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:40.373065 5530 raft_consensus.cc:686] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [term 1 LEADER]: Becoming Leader. State: Replica: b8dc18fbf1954e76a2281d23c744b9e5, State: Running, Role: LEADER
I20260928 08:37:40.373584 5530 consensus_queue.cc:237] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dc18fbf1954e76a2281d23c744b9e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } } peers { permanent_uuid: "f71527a946064a3f8235ac6c970c4a5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41265 } } peers { permanent_uuid: "f786416b6fc04883b4b9ec5b1ebedb9c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33409 } }
I20260928 08:37:40.377823 5058 catalog_manager.cc:5495] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8dc18fbf1954e76a2281d23c744b9e5 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b8dc18fbf1954e76a2281d23c744b9e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dc18fbf1954e76a2281d23c744b9e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f71527a946064a3f8235ac6c970c4a5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f786416b6fc04883b4b9ec5b1ebedb9c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33409 } health_report { overall_health: UNKNOWN } } }
I20260928 08:37:40.407115 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.405012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33686 (local address 127.0.97.2:40391)
0928 08:37:40.405181 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:40.405189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:40.405261 (+ 72us) server_negotiation.cc:407] Connection header received
0928 08:37:40.405491 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:40.405495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:40.405569 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:40.405670 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:40.406480 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:40.406485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:40.406490 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:37:40.406551 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:40.406585 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:37:40.406600 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:40.406725 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:40.406804 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 08:37:40.406860 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 08:37:40.412950 5246 tablet.cc:2369] T e3bf013b67dd4191b0c8bc595e6fcd07 P f786416b6fc04883b4b9ec5b1ebedb9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:37:40.415993 5461 raft_consensus.cc:1261] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b [term 1 FOLLOWER]: Refusing update from remote peer b8dc18fbf1954e76a2281d23c744b9e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:40.415998 5200 raft_consensus.cc:1261] T e3bf013b67dd4191b0c8bc595e6fcd07 P f786416b6fc04883b4b9ec5b1ebedb9c [term 1 FOLLOWER]: Refusing update from remote peer b8dc18fbf1954e76a2281d23c744b9e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:40.416991 5530 consensus_queue.cc:1035] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71527a946064a3f8235ac6c970c4a5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:40.417336 5535 consensus_queue.cc:1035] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f786416b6fc04883b4b9ec5b1ebedb9c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:40.429028 5538 mvcc.cc:205] Tried to move back new op lower bound from 7334234769053356032 to 7334234768894459904. Current Snapshot: MvccSnapshot[applied={T|T < 7334234769053356032}]
I20260928 08:37:40.430455 5539 mvcc.cc:205] Tried to move back new op lower bound from 7334234769053356032 to 7334234768894459904. Current Snapshot: MvccSnapshot[applied={T|T < 7334234769053356032}]
I20260928 08:37:40.444880 5059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:40.445106 5059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:40.448284 5059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eddde00e8f414b8fadd8a58f39a3bb22: Sending DeleteTablet for 3 replicas of tablet e3bf013b67dd4191b0c8bc595e6fcd07
I20260928 08:37:40.449321 5180 tablet_service.cc:1514] Processing DeleteTablet for tablet e3bf013b67dd4191b0c8bc595e6fcd07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:40 UTC) from {username='slave'} at 127.0.0.1:41098
I20260928 08:37:40.449563 5311 tablet_service.cc:1514] Processing DeleteTablet for tablet e3bf013b67dd4191b0c8bc595e6fcd07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:40 UTC) from {username='slave'} at 127.0.0.1:33674
I20260928 08:37:40.449698 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet e3bf013b67dd4191b0c8bc595e6fcd07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:40 UTC) from {username='slave'} at 127.0.0.1:55858
I20260928 08:37:40.451532 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 5118
I20260928 08:37:40.477412 5550 tablet_replica.cc:330] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b: stopping tablet replica
I20260928 08:37:40.478067 5550 raft_consensus.cc:2229] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:40.478770 5550 raft_consensus.cc:2258] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:40.482160 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.452369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41102 (local address 127.0.97.1:33409)
0928 08:37:40.452543 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:40.452549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:40.476279 (+ 23730us) server_negotiation.cc:407] Connection header received
0928 08:37:40.479348 (+ 3069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:40.479358 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:40.479483 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:40.479586 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:40.481113 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:40.481122 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:40.481130 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:37:40.481158 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:40.481210 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:37:40.481232 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:40.481356 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:40.481626 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 08:37:40.481706 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:37:40.483280 5240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:40.483788 5549 tablet_replica.cc:330] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5: stopping tablet replica
I20260928 08:37:40.484347 5549 raft_consensus.cc:2229] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:37:40.485175 5549 raft_consensus.cc:2258] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:40.488021 5550 ts_tablet_manager.cc:1905] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:40.492146 5549 ts_tablet_manager.cc:1905] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:40.496562 5550 ts_tablet_manager.cc:1918] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:37:40.496722 5550 log.cc:1196] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/e3bf013b67dd4191b0c8bc595e6fcd07
I20260928 08:37:40.497263 5550 ts_tablet_manager.cc:1939] T e3bf013b67dd4191b0c8bc595e6fcd07 P f71527a946064a3f8235ac6c970c4a5b: Deleting consensus metadata
I20260928 08:37:40.499119 5046 catalog_manager.cc:4841] TS f71527a946064a3f8235ac6c970c4a5b (127.0.97.3:41265): tablet e3bf013b67dd4191b0c8bc595e6fcd07 (table test-table [id=fc6e1ee3272540d08b5b16c159b07b8a]) successfully deleted
I20260928 08:37:40.500013 5549 ts_tablet_manager.cc:1918] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:37:40.500197 5549 log.cc:1196] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/e3bf013b67dd4191b0c8bc595e6fcd07
I20260928 08:37:40.500643 5549 ts_tablet_manager.cc:1939] T e3bf013b67dd4191b0c8bc595e6fcd07 P b8dc18fbf1954e76a2281d23c744b9e5: Deleting consensus metadata
I20260928 08:37:40.502094 5046 catalog_manager.cc:4841] TS b8dc18fbf1954e76a2281d23c744b9e5 (127.0.97.2:40391): tablet e3bf013b67dd4191b0c8bc595e6fcd07 (table test-table [id=fc6e1ee3272540d08b5b16c159b07b8a]) successfully deleted
I20260928 08:37:40.595970 5548 tablet_replica.cc:330] T e3bf013b67dd4191b0c8bc595e6fcd07 P f786416b6fc04883b4b9ec5b1ebedb9c: stopping tablet replica
I20260928 08:37:40.595989 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 5118
W20260928 08:37:40.613168 5045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33409 (error 108)
I20260928 08:37:40.613428 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 5249
I20260928 08:37:40.615214 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.614747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33409 (local address 127.0.0.1:41112)
0928 08:37:40.614917 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:40.615040 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 08:37:40.615455 5045 catalog_manager.cc:4584] TS f786416b6fc04883b4b9ec5b1ebedb9c (127.0.97.1:33409): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3bf013b67dd4191b0c8bc595e6fcd07: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33409: connect: Connection refused (error 111)
I20260928 08:37:40.616498 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.614306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.0.97.2:40391)
0928 08:37:40.614484 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:40.614489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:40.614541 (+ 52us) server_negotiation.cc:407] Connection header received
0928 08:37:40.614710 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:40.614716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:40.614786 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:40.614890 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:40.615878 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:40.615883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:40.615889 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:37:40.615906 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:40.615946 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:37:40.615962 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:40.616089 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:40.616155 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 08:37:40.616221 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:37:40.617305 5371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:40.669385 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.668818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33409 (local address 127.0.0.1:41124)
0928 08:37:40.669052 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:40.669149 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 08:37:40.670452 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.670095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33409 (local address 127.0.0.1:41128)
0928 08:37:40.670261 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:40.670332 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 08:37:40.725983 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 5249
I20260928 08:37:40.730947 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.730491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33409 (local address 127.0.0.1:41138)
0928 08:37:40.730699 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:40.730802 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 08:37:40.731920 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.731632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33409 (local address 127.0.0.1:41148)
0928 08:37:40.731747 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:40.731808 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 08:37:40.744499 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 5379
I20260928 08:37:40.747174 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.745283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55864 (local address 127.0.97.3:41265)
0928 08:37:40.745459 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:40.745463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:40.745482 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:37:40.745618 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:40.745622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:40.745690 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:40.745777 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:40.746583 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:40.746587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:40.746591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:37:40.746606 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:40.746636 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:37:40.746647 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:40.746748 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:40.746823 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 08:37:40.746868 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:37:40.747900 5501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:40.812451 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.812034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33409 (local address 127.0.0.1:41158)
0928 08:37:40.812221 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:40.812314 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 08:37:40.813509 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:40.813109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33409 (local address 127.0.0.1:41172)
0928 08:37:40.813334 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:40.813399 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 08:37:40.836253 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 5379
I20260928 08:37:40.852456 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 5028
I20260928 08:37:40.853559 5089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:40.957782 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 5028
2026-09-28T08:37:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2570 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 08:37:40.998641 388 mini_oidc.cc:152] Starting JWKS server
I20260928 08:37:41.001157 388 webserver.cc:460] Webserver started at https://127.0.0.1:35719/ using document root <none> and password file <none>
I20260928 08:37:41.001468 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 08:37:41.002090 388 webserver.cc:460] Webserver started at http://127.0.0.1:35689/ using document root <none> and password file <none>
2026-09-28T08:37:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:37:41Z Disabled control of system clock
I20260928 08:37:41.027765 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39225
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39225
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35719/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0/testchainca.cert with env {}
W20260928 08:37:41.348383 5562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:41.348726 5562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:41.348873 5562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:41.356081 5562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:37:41.356220 5562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:41.356307 5562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:41.356408 5562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:37:41.356487 5562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:37:41.356575 5562 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:37:41.356650 5562 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35719/jwks.json
I20260928 08:37:41.366572 5562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39225
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35719/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:41.367868 5562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:41.369145 5562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:41.380785 5568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:41.380911 5567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:41.380878 5570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:41.381767 5562 server_base.cc:1034] running on GCE node
I20260928 08:37:41.382493 5562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:41.383870 5562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:41.385058 5562 hybrid_clock.cc:648] HybridClock initialized: now 1790584661385013 us; error 35 us; skew 500 ppm
I20260928 08:37:41.385411 5562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:37:41.387684 5562 webserver.cc:460] Webserver started at http://127.0.97.62:37899/ using document root <none> and password file <none>
I20260928 08:37:41.388614 5562 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:41.388748 5562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:41.389124 5562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:41.392385 5562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "7586193d0569419e8bb9ed38e5408709"
format_stamp: "Formatted at 2026-09-28 08:37:41 on dist-test-slave-w6c0"
server_key: "f0b22d94db1d291345fcc826aba6814a"
server_key_iv: "5aa16810c29d188f374fd3c8c3080288"
server_key_version: "encryptionkey@0"
I20260928 08:37:41.393153 5562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "7586193d0569419e8bb9ed38e5408709"
format_stamp: "Formatted at 2026-09-28 08:37:41 on dist-test-slave-w6c0"
server_key: "f0b22d94db1d291345fcc826aba6814a"
server_key_iv: "5aa16810c29d188f374fd3c8c3080288"
server_key_version: "encryptionkey@0"
I20260928 08:37:41.398449 5562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:37:41.402012 5576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:41.403620 5562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:37:41.403831 5562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "7586193d0569419e8bb9ed38e5408709"
format_stamp: "Formatted at 2026-09-28 08:37:41 on dist-test-slave-w6c0"
server_key: "f0b22d94db1d291345fcc826aba6814a"
server_key_iv: "5aa16810c29d188f374fd3c8c3080288"
server_key_version: "encryptionkey@0"
I20260928 08:37:41.404065 5562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:41.427503 5562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:41.428459 5562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:41.428787 5562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:41.449499 5562 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39225
I20260928 08:37:41.449524 5627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39225 every 8 connection(s)
I20260928 08:37:41.451006 5562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:37:41.454869 5628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:41.456010 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 5562
I20260928 08:37:41.456243 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:37:41.456890 388 external_mini_cluster.cc:1428] Setting key da9807bef13703396fd6e20c818cab60
I20260928 08:37:41.468621 5628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709: Bootstrap starting.
I20260928 08:37:41.471524 5631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:41.458846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53226 (local address 127.0.97.62:39225)
0928 08:37:41.459894 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:41.459912 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:41.459962 (+ 50us) server_negotiation.cc:407] Connection header received
0928 08:37:41.461596 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:41.461627 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:41.462218 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:41.462648 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:41.463118 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:41.464949 (+ 1831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:41.466350 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:41.467335 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:41.467533 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:41.468730 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:41.468741 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:41.468757 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:37:41.469438 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:41.469530 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:37:41.469561 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:41.469667 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:41.469883 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 08:37:41.470224 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":333,"threads_started":1}
I20260928 08:37:41.472447 5628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:41.473454 5628 log.cc:824] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:41.475314 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:37:41.476254 5628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709: No bootstrap required, opened a new log
I20260928 08:37:41.483346 5628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7586193d0569419e8bb9ed38e5408709" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39225 } }
I20260928 08:37:41.483686 5628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:41.483784 5628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7586193d0569419e8bb9ed38e5408709, State: Initialized, Role: FOLLOWER
I20260928 08:37:41.484352 5628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7586193d0569419e8bb9ed38e5408709" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39225 } }
I20260928 08:37:41.484587 5628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:37:41.484727 5628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:37:41.484885 5628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:41.486977 5628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7586193d0569419e8bb9ed38e5408709" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39225 } }
I20260928 08:37:41.487462 5628 leader_election.cc:302] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7586193d0569419e8bb9ed38e5408709; no voters:
I20260928 08:37:41.488006 5628 leader_election.cc:288] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:37:41.488181 5633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:41.488898 5633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709 [term 1 LEADER]: Becoming Leader. State: Replica: 7586193d0569419e8bb9ed38e5408709, State: Running, Role: LEADER
I20260928 08:37:41.489390 5633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7586193d0569419e8bb9ed38e5408709" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39225 } }
I20260928 08:37:41.490031 5628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:37:41.492658 5634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7586193d0569419e8bb9ed38e5408709" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7586193d0569419e8bb9ed38e5408709" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39225 } } }
I20260928 08:37:41.493050 5634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:41.492650 5635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7586193d0569419e8bb9ed38e5408709. Latest consensus state: current_term: 1 leader_uuid: "7586193d0569419e8bb9ed38e5408709" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7586193d0569419e8bb9ed38e5408709" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39225 } } }
I20260928 08:37:41.493587 5635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:41.495013 5639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:37:41.499465 5639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:37:41.508437 5639 catalog_manager.cc:1347] Generated new cluster ID: fbd9c61265964094ac1d888f9777349b
I20260928 08:37:41.508646 5639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:37:41.518276 5639 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:37:41.519533 5639 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:37:41.533715 5639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7586193d0569419e8bb9ed38e5408709: Generated new TSK 0
I20260928 08:37:41.534596 5639 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 08:37:41.576133 5658 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 08:37:41.576305 5658 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 08:37:41.577236 5631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:41.556000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.0.97.62:39225)
0928 08:37:41.556245 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:41.556252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:41.556650 (+ 398us) server_negotiation.cc:407] Connection header received
0928 08:37:41.556935 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:41.556938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:41.557012 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:41.557129 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:37:41.557749 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:41.558560 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:41.560676 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:41.561128 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:41.561228 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:41.561827 (+ 599us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:37:41.576727 (+ 14900us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35719/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 08:37:41.576999 (+ 272us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53230: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35719/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 08:37:41.577368 5631 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53230: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35719/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 08:37:41.579098 5657 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:37:41.555890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39225 (local address 127.0.0.1:53230)
0928 08:37:41.556530 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:41.556556 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:41.556741 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:41.557179 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:41.557185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:41.557199 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:37:41.557564 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:41.557575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:41.558713 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:41.558720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:41.560499 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:41.560515 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:41.561241 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:41.561285 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:41.561474 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:41.561545 (+ 71us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:37:41.577004 (+ 15459us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35719/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 08:37:41.577353 (+ 349us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39225: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35719/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":315,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:39225: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35719/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 08:37:41.581295 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 5562
I20260928 08:37:41.582340 5623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:41.675143 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 5562
2026-09-28T08:37:41Z chronyd exiting
I20260928 08:37:41.725728 388 test_util.cc:175] -----------------------------------------------
I20260928 08:37:41.725859 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584618551208-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (731 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 08:37:41.729354 388 mini_oidc.cc:152] Starting JWKS server
I20260928 08:37:41.731130 388 webserver.cc:460] Webserver started at https://127.0.0.1:46301/ using document root <none> and password file <none>
I20260928 08:37:41.731307 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 08:37:41.731846 388 webserver.cc:460] Webserver started at http://127.0.0.1:38997/ using document root <none> and password file <none>
2026-09-28T08:37:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:37:41Z Disabled control of system clock
I20260928 08:37:41.757001 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36977
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36977
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46301/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584618551208-388-0/testchainca.cert with env {}
W20260928 08:37:42.073875 5665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:42.074223 5665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:42.074383 5665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:42.081748 5665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:37:42.081897 5665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:42.081982 5665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:42.082082 5665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:37:42.082149 5665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:37:42.082239 5665 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:37:42.082295 5665 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46301/jwks.json
I20260928 08:37:42.092675 5665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36977
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46301/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584618551208-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:42.094043 5665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:42.095336 5665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:42.106722 5673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:42.106765 5671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:42.106797 5670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:42.107630 5665 server_base.cc:1034] running on GCE node
I20260928 08:37:42.108372 5665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:42.109732 5665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:42.110924 5665 hybrid_clock.cc:648] HybridClock initialized: now 1790584662110887 us; error 31 us; skew 500 ppm
I20260928 08:37:42.111268 5665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:37:42.120075 5665 webserver.cc:460] Webserver started at http://127.0.97.62:39797/ using document root <none> and password file <none>
I20260928 08:37:42.120812 5665 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:42.120927 5665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:42.121248 5665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:42.124233 5665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "638403842ce7403b8eac0d2a08ad7356"
format_stamp: "Formatted at 2026-09-28 08:37:42 on dist-test-slave-w6c0"
server_key: "3387127e1f54c08248f566cade84697f"
server_key_iv: "0609479369bc24eb92d400a4a9842dc8"
server_key_version: "encryptionkey@0"
I20260928 08:37:42.125032 5665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "638403842ce7403b8eac0d2a08ad7356"
format_stamp: "Formatted at 2026-09-28 08:37:42 on dist-test-slave-w6c0"
server_key: "3387127e1f54c08248f566cade84697f"
server_key_iv: "0609479369bc24eb92d400a4a9842dc8"
server_key_version: "encryptionkey@0"
I20260928 08:37:42.130041 5665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:37:42.133539 5679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:42.134966 5665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:37:42.135144 5665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "638403842ce7403b8eac0d2a08ad7356"
format_stamp: "Formatted at 2026-09-28 08:37:42 on dist-test-slave-w6c0"
server_key: "3387127e1f54c08248f566cade84697f"
server_key_iv: "0609479369bc24eb92d400a4a9842dc8"
server_key_version: "encryptionkey@0"
I20260928 08:37:42.135356 5665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:42.149748 5665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:42.150636 5665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:42.150904 5665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:42.168718 5665 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36977
I20260928 08:37:42.168764 5730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36977 every 8 connection(s)
I20260928 08:37:42.170059 5665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:37:42.173432 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 5665
I20260928 08:37:42.173627 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:37:42.173816 5731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:42.174300 388 external_mini_cluster.cc:1428] Setting key 19ad3854357eeaa862df4ce0f4ae4355
I20260928 08:37:42.187839 5731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356: Bootstrap starting.
I20260928 08:37:42.188439 5734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:42.176157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.0.97.62:36977)
0928 08:37:42.177142 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:42.177163 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:42.177201 (+ 38us) server_negotiation.cc:407] Connection header received
0928 08:37:42.178703 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:42.178724 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:42.179161 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:42.179540 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:42.180077 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:42.181358 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:42.183070 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:42.184133 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:42.184328 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:42.185708 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:42.185718 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:42.185737 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:37:42.186409 (+ 672us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:42.186542 (+ 133us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:37:42.186573 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:42.186692 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:42.186875 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 08:37:42.187184 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":374,"threads_started":1}
I20260928 08:37:42.191519 5731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:42.192342 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:37:42.192643 5731 log.cc:824] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:42.195564 5731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356: No bootstrap required, opened a new log
I20260928 08:37:42.203152 5731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638403842ce7403b8eac0d2a08ad7356" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36977 } }
I20260928 08:37:42.203490 5731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:42.203589 5731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 638403842ce7403b8eac0d2a08ad7356, State: Initialized, Role: FOLLOWER
I20260928 08:37:42.204087 5731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638403842ce7403b8eac0d2a08ad7356" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36977 } }
I20260928 08:37:42.204283 5731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:37:42.204377 5731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:37:42.204495 5731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:42.206671 5731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638403842ce7403b8eac0d2a08ad7356" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36977 } }
I20260928 08:37:42.207152 5731 leader_election.cc:302] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 638403842ce7403b8eac0d2a08ad7356; no voters:
I20260928 08:37:42.207839 5731 leader_election.cc:288] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:37:42.207988 5736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:42.208730 5736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356 [term 1 LEADER]: Becoming Leader. State: Replica: 638403842ce7403b8eac0d2a08ad7356, State: Running, Role: LEADER
I20260928 08:37:42.209216 5736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638403842ce7403b8eac0d2a08ad7356" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36977 } }
I20260928 08:37:42.209873 5731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:37:42.211898 5737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "638403842ce7403b8eac0d2a08ad7356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638403842ce7403b8eac0d2a08ad7356" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36977 } } }
I20260928 08:37:42.211908 5738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 638403842ce7403b8eac0d2a08ad7356. Latest consensus state: current_term: 1 leader_uuid: "638403842ce7403b8eac0d2a08ad7356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638403842ce7403b8eac0d2a08ad7356" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36977 } } }
I20260928 08:37:42.212345 5738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:42.212342 5737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:42.214150 5741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:37:42.219554 5741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:37:42.228674 5741 catalog_manager.cc:1347] Generated new cluster ID: 23c2b2b7be5846f7b93292be671d2507
I20260928 08:37:42.228787 5741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:37:42.245491 5741 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:37:42.247313 5741 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:37:42.255944 5741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 638403842ce7403b8eac0d2a08ad7356: Generated new TSK 0
I20260928 08:37:42.256896 5741 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 08:37:42.298934 5761 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 08:37:42.299082 5761 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 08:37:42.300099 5734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:42.272651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33406 (local address 127.0.97.62:36977)
0928 08:37:42.272864 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:42.272871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:42.273225 (+ 354us) server_negotiation.cc:407] Connection header received
0928 08:37:42.273454 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:42.273458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:42.273525 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:42.273651 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:37:42.274181 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:42.274939 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:42.276952 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:42.277395 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:42.277494 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:42.278072 (+ 578us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:37:42.299500 (+ 21428us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 08:37:42.299828 (+ 328us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33406: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 08:37:42.300243 5734 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33406: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 08:37:42.300426 5760 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:37:42.272457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36977 (local address 127.0.0.1:33406)
0928 08:37:42.273118 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:42.273138 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:42.273278 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:42.273684 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:42.273690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:42.273703 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:37:42.274018 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:42.274029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:42.275080 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:42.275086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:42.276803 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:42.276817 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:42.277512 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:42.277580 (+ 68us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:42.277769 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:42.277815 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:37:42.299745 (+ 21930us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 08:37:42.300018 (+ 273us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:36977: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46301/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":343,"threads_started":1}
I20260928 08:37:42.301578 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 5665
I20260928 08:37:42.302771 5726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:42.394809 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 5665
2026-09-28T08:37:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (721 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 08:37:42.450908 388 mini_oidc.cc:152] Starting JWKS server
I20260928 08:37:42.452654 388 webserver.cc:460] Webserver started at https://127.0.0.1:43079/ using document root <none> and password file <none>
I20260928 08:37:42.452913 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 08:37:42.453508 388 webserver.cc:460] Webserver started at http://127.0.0.1:33087/ using document root <none> and password file <none>
2026-09-28T08:37:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:37:42Z Disabled control of system clock
I20260928 08:37:42.479396 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34261
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34261
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43079/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 08:37:42.789706 5768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:42.789981 5768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:42.790067 5768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:42.797211 5768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:37:42.797307 5768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:42.797356 5768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:42.797415 5768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:37:42.797458 5768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:37:42.797518 5768 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:37:42.797559 5768 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43079/jwks.json
I20260928 08:37:42.806373 5768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34261
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43079/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:42.807648 5768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:42.809418 5768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:42.820611 5776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:42.820662 5773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:42.820690 5774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:42.821409 5768 server_base.cc:1034] running on GCE node
I20260928 08:37:42.822188 5768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:42.823513 5768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:42.824704 5768 hybrid_clock.cc:648] HybridClock initialized: now 1790584662824663 us; error 31 us; skew 500 ppm
I20260928 08:37:42.825035 5768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:37:42.827294 5768 webserver.cc:460] Webserver started at http://127.0.97.62:36945/ using document root <none> and password file <none>
I20260928 08:37:42.827971 5768 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:42.828058 5768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:42.828370 5768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:42.831202 5768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "13c8b702d9cf4eb7887af8beec5ad978"
format_stamp: "Formatted at 2026-09-28 08:37:42 on dist-test-slave-w6c0"
server_key: "630a4285e0085978666a3a5889c5e8e1"
server_key_iv: "633b9373d06f3e4d13e5387cdf7cce85"
server_key_version: "encryptionkey@0"
I20260928 08:37:42.831981 5768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "13c8b702d9cf4eb7887af8beec5ad978"
format_stamp: "Formatted at 2026-09-28 08:37:42 on dist-test-slave-w6c0"
server_key: "630a4285e0085978666a3a5889c5e8e1"
server_key_iv: "633b9373d06f3e4d13e5387cdf7cce85"
server_key_version: "encryptionkey@0"
I20260928 08:37:42.836987 5768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:37:42.840278 5782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:42.841550 5768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:37:42.841739 5768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "13c8b702d9cf4eb7887af8beec5ad978"
format_stamp: "Formatted at 2026-09-28 08:37:42 on dist-test-slave-w6c0"
server_key: "630a4285e0085978666a3a5889c5e8e1"
server_key_iv: "633b9373d06f3e4d13e5387cdf7cce85"
server_key_version: "encryptionkey@0"
I20260928 08:37:42.841948 5768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:42.856994 5768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:42.857769 5768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:42.858038 5768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:42.881647 5768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34261
I20260928 08:37:42.881669 5833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34261 every 8 connection(s)
I20260928 08:37:42.883476 5768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:37:42.886063 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 5768
I20260928 08:37:42.886273 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:37:42.887132 388 external_mini_cluster.cc:1428] Setting key 492068afca2273524c401072a3efc2cb
I20260928 08:37:42.888373 5834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:42.904102 5837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:42.889039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51812 (local address 127.0.97.62:34261)
0928 08:37:42.890189 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:42.890212 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:42.890266 (+ 54us) server_negotiation.cc:407] Connection header received
0928 08:37:42.893641 (+ 3375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:42.893682 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:42.894281 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:42.894770 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:42.895478 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:42.897282 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:42.899272 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:42.899909 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:42.900105 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:42.901424 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:42.901444 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:42.901464 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:37:42.902293 (+ 829us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:42.902368 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:37:42.902388 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:42.902575 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:42.902839 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 08:37:42.903206 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":360,"threads_started":1}
I20260928 08:37:42.905823 5834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978: Bootstrap starting.
I20260928 08:37:42.907826 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:37:42.910323 5834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:42.911767 5834 log.cc:824] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:42.915285 5834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978: No bootstrap required, opened a new log
I20260928 08:37:42.923158 5834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c8b702d9cf4eb7887af8beec5ad978" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34261 } }
I20260928 08:37:42.923465 5834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:42.923533 5834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c8b702d9cf4eb7887af8beec5ad978, State: Initialized, Role: FOLLOWER
I20260928 08:37:42.924074 5834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c8b702d9cf4eb7887af8beec5ad978" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34261 } }
I20260928 08:37:42.924269 5834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:37:42.924355 5834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:37:42.924491 5834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:42.926442 5834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c8b702d9cf4eb7887af8beec5ad978" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34261 } }
I20260928 08:37:42.926975 5834 leader_election.cc:302] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13c8b702d9cf4eb7887af8beec5ad978; no voters:
I20260928 08:37:42.927534 5834 leader_election.cc:288] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:37:42.927691 5839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:42.928351 5839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978 [term 1 LEADER]: Becoming Leader. State: Replica: 13c8b702d9cf4eb7887af8beec5ad978, State: Running, Role: LEADER
I20260928 08:37:42.928828 5839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c8b702d9cf4eb7887af8beec5ad978" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34261 } }
I20260928 08:37:42.929497 5834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:37:42.931771 5840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13c8b702d9cf4eb7887af8beec5ad978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c8b702d9cf4eb7887af8beec5ad978" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34261 } } }
I20260928 08:37:42.931747 5841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13c8b702d9cf4eb7887af8beec5ad978. Latest consensus state: current_term: 1 leader_uuid: "13c8b702d9cf4eb7887af8beec5ad978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c8b702d9cf4eb7887af8beec5ad978" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34261 } } }
I20260928 08:37:42.932183 5840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:42.932217 5841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:42.933693 5845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:37:42.938913 5845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:37:42.947623 5845 catalog_manager.cc:1347] Generated new cluster ID: 25a851006e2a469596090275131cd838
I20260928 08:37:42.947745 5845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:37:42.966619 5845 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:37:42.967830 5845 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:37:42.976752 5845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13c8b702d9cf4eb7887af8beec5ad978: Generated new TSK 0
I20260928 08:37:42.977523 5845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:37:43.004715 5837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:42.996287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51828 (local address 127.0.97.62:34261)
0928 08:37:42.996482 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:42.996490 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:42.996917 (+ 427us) server_negotiation.cc:407] Connection header received
0928 08:37:42.997145 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:42.997151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:42.997237 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:42.997394 (+ 157us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:37:42.997805 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:42.998976 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:43.000380 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:43.001097 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:43.001225 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:43.001693 (+ 468us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:37:43.003959 (+ 2266us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43079/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 08:37:43.004281 (+ 322us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51828: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43079/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 08:37:43.004841 5837 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51828: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43079/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 08:37:43.006358 5863 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:37:42.996126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34261 (local address 127.0.0.1:51828)
0928 08:37:42.996824 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:42.996840 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:42.996955 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:42.997409 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:42.997413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:42.997422 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:37:42.997674 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:42.997682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:42.999134 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:42.999139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:43.000235 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:43.000246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:43.001227 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:43.001289 (+ 62us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:43.001413 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:43.001445 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:37:43.004160 (+ 2715us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43079/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 08:37:43.005970 (+ 1810us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:34261: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43079/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":1532,"thread_start_us":428,"threads_started":1}
I20260928 08:37:43.007776 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 5768
I20260928 08:37:43.008970 5829 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 08:37:43.017037 5864 webserver.cc:533] Webserver: sslize failed:
E20260928 08:37:43.017122 5864 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 08:37:43.100745 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 5768
2026-09-28T08:37:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (704 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T08:37:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:37:43Z Disabled control of system clock
I20260928 08:37:43.181111 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40955
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40955
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584618551208-388-0/testchainca.cert with env {}
W20260928 08:37:43.509212 5869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:43.509558 5869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:43.509703 5869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:43.516819 5869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:37:43.516957 5869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:43.517050 5869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:43.517160 5869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:37:43.517235 5869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:37:43.517323 5869 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:37:43.517395 5869 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 08:37:43.527136 5869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584618551208-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:43.528496 5869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:43.529745 5869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:43.541831 5875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:43.541886 5874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:43.541910 5877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:43.542585 5869 server_base.cc:1034] running on GCE node
I20260928 08:37:43.543362 5869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:43.544854 5869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:43.546080 5869 hybrid_clock.cc:648] HybridClock initialized: now 1790584663546043 us; error 46 us; skew 500 ppm
I20260928 08:37:43.546429 5869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:37:43.548720 5869 webserver.cc:460] Webserver started at http://127.0.97.62:37537/ using document root <none> and password file <none>
I20260928 08:37:43.549398 5869 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:43.549526 5869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:43.549841 5869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:43.552724 5869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "112a30f838754869872645717be54307"
format_stamp: "Formatted at 2026-09-28 08:37:43 on dist-test-slave-w6c0"
server_key: "de4df63309d5c535f74fde167d65083a"
server_key_iv: "a622f07b7fd847bfb03343663f00698b"
server_key_version: "encryptionkey@0"
I20260928 08:37:43.553507 5869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "112a30f838754869872645717be54307"
format_stamp: "Formatted at 2026-09-28 08:37:43 on dist-test-slave-w6c0"
server_key: "de4df63309d5c535f74fde167d65083a"
server_key_iv: "a622f07b7fd847bfb03343663f00698b"
server_key_version: "encryptionkey@0"
I20260928 08:37:43.558404 5869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 08:37:43.561708 5883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:43.563104 5869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 08:37:43.563385 5869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "112a30f838754869872645717be54307"
format_stamp: "Formatted at 2026-09-28 08:37:43 on dist-test-slave-w6c0"
server_key: "de4df63309d5c535f74fde167d65083a"
server_key_iv: "a622f07b7fd847bfb03343663f00698b"
server_key_version: "encryptionkey@0"
I20260928 08:37:43.563592 5869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:43.592289 5869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:43.593147 5869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:43.593427 5869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:43.611730 5869 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40955
I20260928 08:37:43.611756 5934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40955 every 8 connection(s)
I20260928 08:37:43.613083 5869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:37:43.616863 5935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:43.619923 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 5869
I20260928 08:37:43.620157 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:37:43.620811 388 external_mini_cluster.cc:1428] Setting key f467dc1923ffef1fdd65f43c574f2210
I20260928 08:37:43.628651 5935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307: Bootstrap starting.
I20260928 08:37:43.631949 5935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:43.633016 5935 log.cc:824] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:43.634838 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:43.622663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.0.97.62:40955)
0928 08:37:43.623584 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:43.623597 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:43.623648 (+ 51us) server_negotiation.cc:407] Connection header received
0928 08:37:43.625212 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:43.625232 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:43.625695 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:43.626110 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:43.626703 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:43.627934 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:43.629648 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:43.630693 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:43.630933 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:43.632362 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:43.632374 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:43.632388 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:37:43.632954 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:43.633055 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:37:43.633080 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:43.633220 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:43.633434 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 08:37:43.633814 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":301,"threads_started":1}
I20260928 08:37:43.635977 5935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307: No bootstrap required, opened a new log
I20260928 08:37:43.638496 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:37:43.644573 5935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112a30f838754869872645717be54307" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40955 } }
I20260928 08:37:43.644903 5935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:43.644975 5935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 112a30f838754869872645717be54307, State: Initialized, Role: FOLLOWER
I20260928 08:37:43.645462 5935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112a30f838754869872645717be54307" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40955 } }
I20260928 08:37:43.645650 5935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:37:43.645742 5935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:37:43.645895 5935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:43.648290 5935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112a30f838754869872645717be54307" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40955 } }
I20260928 08:37:43.648730 5935 leader_election.cc:302] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 112a30f838754869872645717be54307; no voters:
I20260928 08:37:43.649350 5935 leader_election.cc:288] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:37:43.649536 5940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:43.650199 5940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307 [term 1 LEADER]: Becoming Leader. State: Replica: 112a30f838754869872645717be54307, State: Running, Role: LEADER
I20260928 08:37:43.650846 5940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112a30f838754869872645717be54307" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40955 } }
I20260928 08:37:43.651566 5935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:37:43.654017 5942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 112a30f838754869872645717be54307. Latest consensus state: current_term: 1 leader_uuid: "112a30f838754869872645717be54307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112a30f838754869872645717be54307" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40955 } } }
I20260928 08:37:43.653990 5941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "112a30f838754869872645717be54307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112a30f838754869872645717be54307" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40955 } } }
I20260928 08:37:43.654438 5942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:43.654495 5941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:43.659926 5945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:37:43.665628 5945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:37:43.674638 5945 catalog_manager.cc:1347] Generated new cluster ID: d26d86cf6bff4bf49c5bbf73568c0985
I20260928 08:37:43.674759 5945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:37:43.693809 5945 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:37:43.695091 5945 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:37:43.703500 5945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 112a30f838754869872645717be54307: Generated new TSK 0
I20260928 08:37:43.704493 5945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:37:43.842898 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:43.723843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.0.97.62:40955)
0928 08:37:43.724050 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:43.724060 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:43.724545 (+ 485us) server_negotiation.cc:407] Connection header received
0928 08:37:43.724760 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:43.724764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:43.724837 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:43.724955 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:37:43.725457 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:43.726179 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:43.728208 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:43.728645 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:43.728741 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:43.729283 (+ 542us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:37:43.842242 (+112959us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 08:37:43.842653 (+ 411us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40278: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 08:37:43.843060 5938 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40278: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 08:37:43.843299 5964 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:37:43.723830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40955 (local address 127.0.0.1:40278)
0928 08:37:43.724441 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:43.724463 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:43.724607 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:43.724999 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:43.725004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:43.725015 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:37:43.725318 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:43.725332 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:43.726321 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:43.726328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:43.728080 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:43.728093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:43.728759 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:43.728814 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:43.729026 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:43.729071 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:37:43.842566 (+113495us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 08:37:43.842881 (+ 315us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:40955: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":319,"threads_started":1}
I20260928 08:37:43.844882 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 5869
I20260928 08:37:43.846040 5930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:43.948891 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 5869
2026-09-28T08:37:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (828 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (405 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1486 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:37:45 dist-test-slave-w6c0 krb5kdc[5980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:37:45 dist-test-slave-w6c0 krb5kdc[5980](info): set up 2 sockets
Sep 28 08:37:45 dist-test-slave-w6c0 krb5kdc[5980](info): commencing operation
krb5kdc: starting...
W20260928 08:37:46.935406 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:37:47 dist-test-slave-w6c0 krb5kdc[5980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:37:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:37:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:37:47.147208 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45095
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:47.487260 5993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:47.487603 5993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:47.487776 5993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:47.495622 5993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:37:47.495816 5993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:47.495936 5993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:47.496048 5993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:37:47.496130 5993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:37:47.507115 5993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:47.508534 5993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:47.509876 5993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:47.523769 6001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:47.523890 5999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:47.524067 5998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:47.524614 5993 server_base.cc:1034] running on GCE node
I20260928 08:37:47.525421 5993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:47.527184 5993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:47.528450 5993 hybrid_clock.cc:648] HybridClock initialized: now 1790584667528390 us; error 60 us; skew 500 ppm
I20260928 08:37:47.528853 5993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:47 dist-test-slave-w6c0 krb5kdc[5980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:47.536990 5993 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:37:47.538759 5993 webserver.cc:460] Webserver started at http://127.0.97.62:35207/ using document root <none> and password file <none>
I20260928 08:37:47.539497 5993 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:47.539659 5993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:47.539984 5993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:47.543110 5993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "0284adf7318448f1882706c5b2055365"
format_stamp: "Formatted at 2026-09-28 08:37:47 on dist-test-slave-w6c0"
server_key: "9de902b097995dca20a8a85bd43c1b95"
server_key_iv: "4705b52be1cc47cceaa1029f104bd537"
server_key_version: "encryptionkey@0"
I20260928 08:37:47.543910 5993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "0284adf7318448f1882706c5b2055365"
format_stamp: "Formatted at 2026-09-28 08:37:47 on dist-test-slave-w6c0"
server_key: "9de902b097995dca20a8a85bd43c1b95"
server_key_iv: "4705b52be1cc47cceaa1029f104bd537"
server_key_version: "encryptionkey@0"
I20260928 08:37:47.549189 5993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:37:47.552718 6008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:47.554278 5993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:37:47.554489 5993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "0284adf7318448f1882706c5b2055365"
format_stamp: "Formatted at 2026-09-28 08:37:47 on dist-test-slave-w6c0"
server_key: "9de902b097995dca20a8a85bd43c1b95"
server_key_iv: "4705b52be1cc47cceaa1029f104bd537"
server_key_version: "encryptionkey@0"
I20260928 08:37:47.554777 5993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:47.574115 5993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:47.578903 5993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:47.579223 5993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:47.598841 5993 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45095
I20260928 08:37:47.598883 6059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45095 every 8 connection(s)
I20260928 08:37:47.600308 5993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:37:47.604154 6060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:47.608204 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 5993
I20260928 08:37:47.608443 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:37:47.609252 388 external_mini_cluster.cc:1428] Setting key b7c3289abdb377e00a828271fe1631bf
I20260928 08:37:47.618806 6060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365: Bootstrap starting.
I20260928 08:37:47.621869 6060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:47.623052 6060 log.cc:824] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365: Log is configured to *not* fsync() on all Append() calls
Sep 28 08:37:47 dist-test-slave-w6c0 krb5kdc[5980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:47.626065 6060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365: No bootstrap required, opened a new log
I20260928 08:37:47.635293 6063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:47.611840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45434 (local address 127.0.97.62:45095)
0928 08:37:47.612795 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:47.612822 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:47.612884 (+ 62us) server_negotiation.cc:407] Connection header received
0928 08:37:47.615216 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:47.615247 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:47.615759 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:47.616233 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:47.617966 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:47.619448 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:47.620705 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:47.621358 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:47.621552 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:47.625929 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:47.625940 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:47.625957 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:47.626002 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:47.629987 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:47.631141 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:47.631149 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:47.631159 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:47.631390 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:47.631897 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:47.631902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:47.631906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:47.632629 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:47.632947 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:47.633411 (+ 464us) server_negotiation.cc:299] Negotiation successful
0928 08:37:47.633871 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":391,"threads_started":1}
I20260928 08:37:47.636796 6060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0284adf7318448f1882706c5b2055365" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45095 } }
I20260928 08:37:47.637190 6060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:47.637317 6060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0284adf7318448f1882706c5b2055365, State: Initialized, Role: FOLLOWER
I20260928 08:37:47.637897 6060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0284adf7318448f1882706c5b2055365" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45095 } }
I20260928 08:37:47.638120 6060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:37:47.638274 6060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:37:47.638455 6060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:47.640586 6060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0284adf7318448f1882706c5b2055365" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45095 } }
I20260928 08:37:47.641063 6060 leader_election.cc:302] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0284adf7318448f1882706c5b2055365; no voters:
I20260928 08:37:47.641623 6060 leader_election.cc:288] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:37:47.641785 6065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:47.642525 6065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365 [term 1 LEADER]: Becoming Leader. State: Replica: 0284adf7318448f1882706c5b2055365, State: Running, Role: LEADER
I20260928 08:37:47.643108 6065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0284adf7318448f1882706c5b2055365" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45095 } }
I20260928 08:37:47.643627 6060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:37:47.646391 6067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0284adf7318448f1882706c5b2055365. Latest consensus state: current_term: 1 leader_uuid: "0284adf7318448f1882706c5b2055365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0284adf7318448f1882706c5b2055365" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45095 } } }
I20260928 08:37:47.646327 6066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0284adf7318448f1882706c5b2055365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0284adf7318448f1882706c5b2055365" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45095 } } }
I20260928 08:37:47.647087 6066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:47.647804 6067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365 [sys.catalog]: This master's current role is: LEADER
W20260928 08:37:47.654575 6080 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:37:47.654693 6080 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:37:47.654835 6074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:37:47.657025 6074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:37:47.666229 6074 catalog_manager.cc:1347] Generated new cluster ID: 006393c6743f42c7b1cf6ab8150fe7e1
I20260928 08:37:47.666356 6074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:37:47.699436 6074 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:37:47.700920 6074 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:37:47.716503 6074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0284adf7318448f1882706c5b2055365: Generated new TSK 0
I20260928 08:37:47.717562 6074 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:37:47.775816 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45095
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:48.099186 6086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:48.099454 6086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:48.099611 6086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:48.106925 6086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:48.107056 6086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:48.107228 6086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:37:48.116822 6086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:48.118089 6086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:48.119390 6086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:48.132263 6094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:48.132381 6091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:48.132331 6092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:48.133318 6086 server_base.cc:1034] running on GCE node
I20260928 08:37:48.133997 6086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:48.135056 6086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:48.136313 6086 hybrid_clock.cc:648] HybridClock initialized: now 1790584668136232 us; error 69 us; skew 500 ppm
I20260928 08:37:48.136701 6086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:48 dist-test-slave-w6c0 krb5kdc[5980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:48.140704 6086 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:37:48.142069 6086 webserver.cc:460] Webserver started at http://127.0.97.1:37075/ using document root <none> and password file <none>
I20260928 08:37:48.142831 6086 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:48.142974 6086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:48.143277 6086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:48.146337 6086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "3da390a272ef4edebdbd653bb14b11b4"
format_stamp: "Formatted at 2026-09-28 08:37:48 on dist-test-slave-w6c0"
server_key: "962f4f528ea801e82fcd9c87f8302648"
server_key_iv: "a73b27f1159fc5a9f7b7b2d33a3e1578"
server_key_version: "encryptionkey@0"
I20260928 08:37:48.147171 6086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3da390a272ef4edebdbd653bb14b11b4"
format_stamp: "Formatted at 2026-09-28 08:37:48 on dist-test-slave-w6c0"
server_key: "962f4f528ea801e82fcd9c87f8302648"
server_key_iv: "a73b27f1159fc5a9f7b7b2d33a3e1578"
server_key_version: "encryptionkey@0"
I20260928 08:37:48.152400 6086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:37:48.156056 6101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:48.157395 6086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:37:48.157564 6086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "3da390a272ef4edebdbd653bb14b11b4"
format_stamp: "Formatted at 2026-09-28 08:37:48 on dist-test-slave-w6c0"
server_key: "962f4f528ea801e82fcd9c87f8302648"
server_key_iv: "a73b27f1159fc5a9f7b7b2d33a3e1578"
server_key_version: "encryptionkey@0"
I20260928 08:37:48.157775 6086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:48.193430 6086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:48.198412 6086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:48.198745 6086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:48.199709 6086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:48.201028 6086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:48.201138 6086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:48.201282 6086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:48.201355 6086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:48.237690 6086 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44183
I20260928 08:37:48.237778 6213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44183 every 8 connection(s)
I20260928 08:37:48.239037 6086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:37:48.246084 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 6086
I20260928 08:37:48.246281 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:37:48.247117 388 external_mini_cluster.cc:1428] Setting key bc056578a4822bc205e7b6add21a0c62
Sep 28 08:37:48 dist-test-slave-w6c0 krb5kdc[5980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:48.262917 6216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:48.241716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55755 (local address 127.0.97.62:45095)
0928 08:37:48.242202 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:48.242207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:48.243182 (+ 975us) server_negotiation.cc:407] Connection header received
0928 08:37:48.245060 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:48.245065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:48.245146 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:48.245244 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:48.246839 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:48.247895 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:48.250064 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:48.250489 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:48.250678 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:48.255668 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:48.255673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:48.255676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:48.255708 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:48.259525 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:48.260328 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:48.260336 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:48.260343 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:48.260485 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:48.260909 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:48.260917 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:48.260923 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:48.261288 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:48.261435 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:48.262149 (+ 714us) server_negotiation.cc:299] Negotiation successful
0928 08:37:48.262396 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":253,"threads_started":1}
I20260928 08:37:48.263685 6217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:48.242223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45095 (local address 127.0.97.1:55755)
0928 08:37:48.243016 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:48.243069 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:48.244771 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:48.245436 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:48.245443 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:48.245940 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:48.246670 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:48.246689 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:48.248064 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:48.248070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:48.249884 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:48.249901 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:48.250780 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:48.250791 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:48.251129 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:48.252057 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:48.252089 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:48.255443 (+ 3354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:48.259664 (+ 4221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:48.259673 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:48.259687 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:48.260123 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:48.260593 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:48.260596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:48.260598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:48.260776 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:48.261423 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:48.261430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:48.261753 (+ 323us) client_negotiation.cc:769] Sending connection context
0928 08:37:48.261979 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 08:37:48.262497 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":363,"threads_started":1}
I20260928 08:37:48.265271 6214 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45095
I20260928 08:37:48.265585 6214 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:48.266381 6214 heartbeater.cc:511] Master 127.0.97.62:45095 requested a full tablet report, sending...
I20260928 08:37:48.268395 6025 ts_manager.cc:194] Registered new tserver with Master: 3da390a272ef4edebdbd653bb14b11b4 (127.0.97.1:44183)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:37:48.269852 6025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:37:48.298446 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45095
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:48.617775 6220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:48.618046 6220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:48.618207 6220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:48.625458 6220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:48.625563 6220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:48.625737 6220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:37:48.635422 6220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:48.636757 6220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:48.638027 6220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:48.651558 6228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:48.651691 6225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:48.651887 6226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:48.653262 6220 server_base.cc:1034] running on GCE node
I20260928 08:37:48.654170 6220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:48.655638 6220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:48.656898 6220 hybrid_clock.cc:648] HybridClock initialized: now 1790584668656838 us; error 43 us; skew 500 ppm
I20260928 08:37:48.657351 6220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:48 dist-test-slave-w6c0 krb5kdc[5980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:48.660780 6220 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:37:48.662434 6220 webserver.cc:460] Webserver started at http://127.0.97.2:45381/ using document root <none> and password file <none>
I20260928 08:37:48.663165 6220 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:48.663304 6220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:48.663614 6220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:48.666635 6220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "4c86580643b14fde8e56804321cda9ba"
format_stamp: "Formatted at 2026-09-28 08:37:48 on dist-test-slave-w6c0"
server_key: "0f116f17d3cd66c9aca09c7a597f782b"
server_key_iv: "05decb15d6f3d6564bbd9811bd6ef394"
server_key_version: "encryptionkey@0"
I20260928 08:37:48.667421 6220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4c86580643b14fde8e56804321cda9ba"
format_stamp: "Formatted at 2026-09-28 08:37:48 on dist-test-slave-w6c0"
server_key: "0f116f17d3cd66c9aca09c7a597f782b"
server_key_iv: "05decb15d6f3d6564bbd9811bd6ef394"
server_key_version: "encryptionkey@0"
I20260928 08:37:48.672534 6220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:37:48.676189 6235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:48.677659 6220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:37:48.677850 6220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "4c86580643b14fde8e56804321cda9ba"
format_stamp: "Formatted at 2026-09-28 08:37:48 on dist-test-slave-w6c0"
server_key: "0f116f17d3cd66c9aca09c7a597f782b"
server_key_iv: "05decb15d6f3d6564bbd9811bd6ef394"
server_key_version: "encryptionkey@0"
I20260928 08:37:48.678072 6220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:48.694046 6220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:48.697945 6220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:48.698243 6220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:48.699282 6220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:48.700733 6220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:48.700832 6220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:48.700987 6220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:48.701058 6220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:48.735352 6220 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35369
I20260928 08:37:48.735395 6347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35369 every 8 connection(s)
I20260928 08:37:48.736624 6220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:37:48.738437 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 6220
I20260928 08:37:48.738624 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:37:48.739209 388 external_mini_cluster.cc:1428] Setting key 253b453df9e74ce3868ab65073555201
Sep 28 08:37:48 dist-test-slave-w6c0 krb5kdc[5980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:48.760692 6216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:48.739243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38359 (local address 127.0.97.62:45095)
0928 08:37:48.739452 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:48.739458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:48.740844 (+ 1386us) server_negotiation.cc:407] Connection header received
0928 08:37:48.743552 (+ 2708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:48.743556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:48.743706 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:48.743791 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:48.745870 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:48.746652 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:48.748896 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:48.749306 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:48.749447 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:48.753007 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:48.753016 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:48.753023 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:48.753068 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:48.756681 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:48.757912 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:48.757919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:48.757925 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:48.758090 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:48.758709 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:48.758717 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:48.758724 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:48.759170 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:48.759327 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:48.760054 (+ 727us) server_negotiation.cc:299] Negotiation successful
0928 08:37:48.760308 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 08:37:48.761554 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:48.739819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45095 (local address 127.0.97.2:38359)
0928 08:37:48.740648 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:48.740719 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:48.743258 (+ 2539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:48.744114 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:48.744128 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:48.744887 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:48.745683 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:48.745710 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:48.746806 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:48.746814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:48.748732 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:48.748746 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:48.749542 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:48.749551 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:48.749784 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:48.750638 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:48.750662 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:48.752772 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:48.756976 (+ 4204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:48.756995 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:48.757033 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:48.757789 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:48.758304 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:48.758308 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:48.758311 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:48.758597 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:48.759322 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:48.759329 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:48.759673 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 08:37:48.759910 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 08:37:48.760335 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":389,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 08:37:48.763388 6348 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45095
I20260928 08:37:48.763741 6348 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:48.764700 6348 heartbeater.cc:511] Master 127.0.97.62:45095 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 08:37:48.766214 6025 ts_manager.cc:194] Registered new tserver with Master: 4c86580643b14fde8e56804321cda9ba (127.0.97.2:35369)
I20260928 08:37:48.767112 6025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:37:48.790704 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45095
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:49.141350 6353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:49.141624 6353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:49.141718 6353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:49.149564 6353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:49.149675 6353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:49.149791 6353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:37:49.159730 6353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:49.161048 6353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:49.162362 6353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:49.175429 6358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:49.175483 6359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:49.175637 6361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:49.176285 6353 server_base.cc:1034] running on GCE node
I20260928 08:37:49.176955 6353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:49.177893 6353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:49.179111 6353 hybrid_clock.cc:648] HybridClock initialized: now 1790584669179066 us; error 42 us; skew 500 ppm
I20260928 08:37:49.179415 6353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:49 dist-test-slave-w6c0 krb5kdc[5980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:49.182617 6353 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:37:49.184093 6353 webserver.cc:460] Webserver started at http://127.0.97.3:45795/ using document root <none> and password file <none>
I20260928 08:37:49.184774 6353 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:49.184919 6353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:49.185222 6353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:49.188150 6353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "ac82b695c31b4e60a7323a5c3c470a86"
format_stamp: "Formatted at 2026-09-28 08:37:49 on dist-test-slave-w6c0"
server_key: "bf063b1c49adee78e3d9728e064aafb8"
server_key_iv: "6e48ab3d923f86bff799cf71341c3963"
server_key_version: "encryptionkey@0"
I20260928 08:37:49.188930 6353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ac82b695c31b4e60a7323a5c3c470a86"
format_stamp: "Formatted at 2026-09-28 08:37:49 on dist-test-slave-w6c0"
server_key: "bf063b1c49adee78e3d9728e064aafb8"
server_key_iv: "6e48ab3d923f86bff799cf71341c3963"
server_key_version: "encryptionkey@0"
I20260928 08:37:49.194221 6353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:37:49.199704 6368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:49.201339 6353 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 08:37:49.201470 6353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "ac82b695c31b4e60a7323a5c3c470a86"
format_stamp: "Formatted at 2026-09-28 08:37:49 on dist-test-slave-w6c0"
server_key: "bf063b1c49adee78e3d9728e064aafb8"
server_key_iv: "6e48ab3d923f86bff799cf71341c3963"
server_key_version: "encryptionkey@0"
I20260928 08:37:49.201648 6353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:49.215669 6353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:49.219584 6353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:49.219835 6353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:49.220742 6353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:49.222010 6353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:49.222083 6353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:49.222163 6353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:49.222203 6353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:49.260675 6353 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32921
I20260928 08:37:49.260756 6480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32921 every 8 connection(s)
I20260928 08:37:49.262223 6353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:37:49.271569 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 6353
I20260928 08:37:49.271775 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:37:49.272243 388 external_mini_cluster.cc:1428] Setting key 952c11366387c452c9f358a42c608592
I20260928 08:37:49.272895 6214 heartbeater.cc:503] Master 127.0.97.62:45095 was elected leader, sending a full tablet report...
Sep 28 08:37:49 dist-test-slave-w6c0 krb5kdc[5980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:37:49.284912 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:49.265076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33213 (local address 127.0.97.62:45095)
0928 08:37:49.265643 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:49.265649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:49.266561 (+ 912us) server_negotiation.cc:407] Connection header received
0928 08:37:49.268286 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:49.268290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:49.268368 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:49.268469 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:49.269989 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:49.271078 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:49.273649 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:49.274124 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:49.274267 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:49.279104 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:49.279110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:49.279113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:49.279157 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:49.281710 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:49.282533 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:49.282541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:49.282547 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:49.282677 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:49.283106 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:49.283114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:49.283117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:49.283399 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:49.283574 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:49.284300 (+ 726us) server_negotiation.cc:299] Negotiation successful
0928 08:37:49.284533 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":280,"threads_started":1}
I20260928 08:37:49.285890 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:49.265625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45095 (local address 127.0.97.3:33213)
0928 08:37:49.266375 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:49.266415 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:49.268031 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:49.268668 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:49.268675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:49.269184 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:49.269710 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:49.269726 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:49.271515 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:49.271520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:49.273466 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:49.273489 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:49.274404 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:49.274412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:49.274797 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:49.275934 (+ 1137us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:49.275964 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:49.278855 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:49.281916 (+ 3061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:49.281924 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:49.281938 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:49.282364 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:49.282790 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:49.282795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:49.282798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:49.282994 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:49.283552 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:49.283561 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:49.283930 (+ 369us) client_negotiation.cc:769] Sending connection context
0928 08:37:49.284211 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 08:37:49.284768 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":322,"threads_started":1}
I20260928 08:37:49.287304 6481 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45095
I20260928 08:37:49.287667 6481 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:49.288552 6481 heartbeater.cc:511] Master 127.0.97.62:45095 requested a full tablet report, sending...
I20260928 08:37:49.289964 6024 ts_manager.cc:194] Registered new tserver with Master: ac82b695c31b4e60a7323a5c3c470a86 (127.0.97.3:32921)
I20260928 08:37:49.290936 6024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33213
I20260928 08:37:49.301499 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/test-admin.keytab.
Sep 28 08:37:49 dist-test-slave-w6c0 krb5kdc[5980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:49.328190 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 08:37:49.333402 6489 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:37:49.330992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56570 (local address 127.0.97.1:44183)
0928 08:37:49.331594 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:49.331601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:49.331650 (+ 49us) server_negotiation.cc:407] Connection header received
0928 08:37:49.331741 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:49.331747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:49.331979 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:49.332109 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:49.333019 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56570: BlockingRecv error: recv got EOF from 127.0.0.1:56570 (error 108)
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":317,"threads_started":1}
W20260928 08:37:49.333897 388 init.cc:244] Error while opening credential cache '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/krb5cc)
Sep 28 08:37:49 dist-test-slave-w6c0 krb5kdc[5980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:49.335538 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 08:37:49 dist-test-slave-w6c0 krb5kdc[5980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:49.336862 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 08:37:49 dist-test-slave-w6c0 krb5kdc[5980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:37:49.351775 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:49.338362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.0.97.1:44183)
0928 08:37:49.338591 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:49.338598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:49.338822 (+ 224us) server_negotiation.cc:407] Connection header received
0928 08:37:49.338967 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:49.338979 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:49.339038 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:49.339119 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:49.340091 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:49.341089 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:49.342285 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:49.342924 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:49.343053 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:49.346483 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:49.346493 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:49.346550 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:49.346602 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:49.349317 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:49.349900 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:49.349907 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:49.349909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:49.350041 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:49.350345 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:49.350348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:49.350349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:49.350738 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:49.350940 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:49.351241 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 08:37:49.351502 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 08:37:49.357550 6489 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:37:49.355854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56588 (local address 127.0.97.1:44183)
0928 08:37:49.356087 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:49.356094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:49.356280 (+ 186us) server_negotiation.cc:407] Connection header received
0928 08:37:49.356450 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:49.356462 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:49.356515 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:49.356613 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:49.357290 (+ 677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56588: BlockingRecv error: recv got EOF from 127.0.0.1:56588 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260928 08:37:49.357721 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/krb5cc' of type FILE
Sep 28 08:37:49 dist-test-slave-w6c0 krb5kdc[5980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:49.359282 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 08:37:49 dist-test-slave-w6c0 krb5kdc[5980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:49.360565 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 08:37:49 dist-test-slave-w6c0 krb5kdc[5980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:37:49.373991 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:49.361861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56594 (local address 127.0.97.1:44183)
0928 08:37:49.362104 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:49.362111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:49.362318 (+ 207us) server_negotiation.cc:407] Connection header received
0928 08:37:49.362490 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:49.362496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:49.362585 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:49.362682 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:49.363593 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:49.364283 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:49.365462 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:49.365900 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:49.366050 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:49.369273 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:49.369279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:49.369284 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:49.369322 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:49.371618 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:49.372227 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:49.372235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:49.372238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:49.372357 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:49.372715 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:49.372720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:49.372724 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:49.372967 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:49.373220 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:49.373513 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 08:37:49.373735 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 08:37:49.378082 6489 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:37:49.376444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.0.97.1:44183)
0928 08:37:49.376646 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:49.376653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:49.376867 (+ 214us) server_negotiation.cc:407] Connection header received
0928 08:37:49.377042 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:49.377047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:49.377119 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:49.377215 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:49.377861 (+ 646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56606: BlockingRecv error: recv got EOF from 127.0.0.1:56606 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 08:37:49.378278 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584618551208-388-0/krb5kdc/krb5cc' of type FILE
Sep 28 08:37:49 dist-test-slave-w6c0 krb5kdc[5980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:49.379770 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 08:37:49 dist-test-slave-w6c0 krb5kdc[5980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:49.381028 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 08:37:49 dist-test-slave-w6c0 krb5kdc[5980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:37:49.395277 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:49.382417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56616 (local address 127.0.97.1:44183)
0928 08:37:49.382649 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:49.382657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:49.382929 (+ 272us) server_negotiation.cc:407] Connection header received
0928 08:37:49.383110 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:49.383115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:49.383167 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:49.383265 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:49.384207 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:49.384899 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:49.386076 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:49.386493 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:49.386669 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:49.390004 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:49.390009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:49.390012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:49.390043 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:49.392700 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:49.393368 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:49.393373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:49.393391 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:49.393534 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:49.393886 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:49.393893 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:49.393899 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:49.394155 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:49.394487 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:49.394759 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 08:37:49.394962 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 08:37:49.396855 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 6086
I20260928 08:37:49.408375 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:49.397508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.0.97.1:44183)
0928 08:37:49.397686 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:49.397690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:49.398049 (+ 359us) server_negotiation.cc:407] Connection header received
0928 08:37:49.398156 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:49.398160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:49.398214 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:49.398287 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:49.399251 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:49.399922 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:49.401173 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:49.401593 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:49.401691 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:49.403733 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:49.403738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:49.403741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:49.403769 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:49.406187 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:49.406809 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:49.406813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:49.406815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:49.406939 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:49.407252 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:49.407255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:49.407257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:49.407491 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:49.407673 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:49.407904 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 08:37:49.408115 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 08:37:49.408959 6209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:49.493409 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 6086
I20260928 08:37:49.510763 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 6220
Sep 28 08:37:49 dist-test-slave-w6c0 krb5kdc[5980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:37:49.525547 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:49.511632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37312 (local address 127.0.97.2:35369)
0928 08:37:49.512216 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:49.512223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:49.512249 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:37:49.512313 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:49.512319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:49.512529 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:49.512659 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:49.513537 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:49.514725 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:49.515915 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:49.516439 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:49.516542 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:49.520114 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:49.520124 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:49.520137 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:49.520166 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:49.523058 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:49.523654 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:49.523661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:49.523663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:49.523769 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:49.524129 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:49.524133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:49.524135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:49.524494 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:49.524727 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:49.524963 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 08:37:49.525223 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":306,"threads_started":1}
I20260928 08:37:49.526827 6343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:49.609534 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 6220
I20260928 08:37:49.625358 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 6353
Sep 28 08:37:49 dist-test-slave-w6c0 krb5kdc[5980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:37:49.640697 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:49.626548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51574 (local address 127.0.97.3:32921)
0928 08:37:49.627219 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:49.627226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:49.627255 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:37:49.627315 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:49.627321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:49.627536 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:49.627663 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:49.628624 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:49.629649 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:49.630941 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:49.631473 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:49.631570 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:49.635355 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:49.635365 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:49.635379 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:49.635408 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:49.638168 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:49.638907 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:49.638914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:49.638916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:49.639042 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:49.639362 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:49.639365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:49.639367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:49.639682 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:49.639933 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:49.640138 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 08:37:49.640392 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":365,"threads_started":1}
I20260928 08:37:49.642138 6476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:49.725178 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 6353
I20260928 08:37:49.741658 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 5993
I20260928 08:37:49.742820 6055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:49.845654 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 5993
2026-09-28T08:37:49Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4010 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:37:49 dist-test-slave-w6c0 krb5kdc[6509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:37:49 dist-test-slave-w6c0 krb5kdc[6509](info): set up 2 sockets
Sep 28 08:37:49 dist-test-slave-w6c0 krb5kdc[6509](info): commencing operation
krb5kdc: starting...
W20260928 08:37:50.943447 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:37:51 dist-test-slave-w6c0 krb5kdc[6509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:37:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:37:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260928 08:37:51.157816 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42403
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:51.485780 6522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:51.486117 6522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:51.486294 6522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:51.493660 6522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:37:51.493824 6522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:51.493911 6522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:51.494014 6522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:37:51.494081 6522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:37:51.504338 6522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:51.505722 6522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:51.507102 6522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:51.519214 6527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:51.519294 6528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:51.519310 6530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:51.520313 6522 server_base.cc:1034] running on GCE node
I20260928 08:37:51.521095 6522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:51.522454 6522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:51.523736 6522 hybrid_clock.cc:648] HybridClock initialized: now 1790584671523691 us; error 35 us; skew 500 ppm
I20260928 08:37:51.524097 6522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:51 dist-test-slave-w6c0 krb5kdc[6509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584671, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:51.527144 6522 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260928 08:37:51.528489 6522 webserver.cc:460] Webserver started at http://127.0.97.62:44081/ using document root <none> and password file <none>
I20260928 08:37:51.529109 6522 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:51.529204 6522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:51.529438 6522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:51.532302 6522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "899270827952449d802397379a7ed6ef"
format_stamp: "Formatted at 2026-09-28 08:37:51 on dist-test-slave-w6c0"
server_key: "8d1d62cbbe772f02dc720d6cb24b138a"
server_key_iv: "b054570e2b2561a0c9b36367a258fc9c"
server_key_version: "encryptionkey@0"
I20260928 08:37:51.532989 6522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "899270827952449d802397379a7ed6ef"
format_stamp: "Formatted at 2026-09-28 08:37:51 on dist-test-slave-w6c0"
server_key: "8d1d62cbbe772f02dc720d6cb24b138a"
server_key_iv: "b054570e2b2561a0c9b36367a258fc9c"
server_key_version: "encryptionkey@0"
I20260928 08:37:51.537822 6522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:37:51.541035 6537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:51.542325 6522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:37:51.542498 6522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "899270827952449d802397379a7ed6ef"
format_stamp: "Formatted at 2026-09-28 08:37:51 on dist-test-slave-w6c0"
server_key: "8d1d62cbbe772f02dc720d6cb24b138a"
server_key_iv: "b054570e2b2561a0c9b36367a258fc9c"
server_key_version: "encryptionkey@0"
I20260928 08:37:51.542695 6522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:51.568375 6522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:51.572170 6522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:51.572395 6522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:51.590377 6522 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42403
I20260928 08:37:51.590416 6588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42403 every 8 connection(s)
I20260928 08:37:51.591771 6522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:37:51.595364 6589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:51.598099 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 6522
I20260928 08:37:51.598308 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:37:51.598980 388 external_mini_cluster.cc:1428] Setting key a73748e1945d0528f6582746986139a0
I20260928 08:37:51.607117 6589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef: Bootstrap starting.
I20260928 08:37:51.610023 6589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:51.611038 6589 log.cc:824] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef: Log is configured to *not* fsync() on all Append() calls
Sep 28 08:37:51 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:37:51.613971 6589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef: No bootstrap required, opened a new log
I20260928 08:37:51.621335 6592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:51.600915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43230 (local address 127.0.97.62:42403)
0928 08:37:51.601733 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:51.601746 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:51.601789 (+ 43us) server_negotiation.cc:407] Connection header received
0928 08:37:51.603462 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:51.603484 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:51.603874 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:51.604253 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:51.605619 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:51.606928 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:51.608158 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:51.608821 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:51.609009 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:51.612952 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:51.612962 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:51.612980 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:51.613027 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:51.616919 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:51.617571 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:51.617584 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:51.617592 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:51.617788 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:51.618195 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:51.618200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:51.618203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:51.618898 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:51.619188 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:51.619531 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 08:37:51.619988 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":362,"threads_started":1}
I20260928 08:37:51.624377 6589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } }
I20260928 08:37:51.624681 6589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:51.624756 6589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 899270827952449d802397379a7ed6ef, State: Initialized, Role: FOLLOWER
I20260928 08:37:51.625280 6589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } }
I20260928 08:37:51.625465 6589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:37:51.625557 6589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:37:51.625679 6589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:51.627851 6589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } }
I20260928 08:37:51.628448 6589 leader_election.cc:302] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 899270827952449d802397379a7ed6ef; no voters:
I20260928 08:37:51.629032 6589 leader_election.cc:288] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:37:51.629204 6594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:51.629935 6594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [term 1 LEADER]: Becoming Leader. State: Replica: 899270827952449d802397379a7ed6ef, State: Running, Role: LEADER
I20260928 08:37:51.630426 6594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } }
I20260928 08:37:51.631018 6589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:37:51.633308 6596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 899270827952449d802397379a7ed6ef. Latest consensus state: current_term: 1 leader_uuid: "899270827952449d802397379a7ed6ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } }
I20260928 08:37:51.633709 6596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:51.634310 6595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "899270827952449d802397379a7ed6ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } }
I20260928 08:37:51.634863 6595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:51.635964 6599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:37:51.641978 6599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:37:51.650377 6599 catalog_manager.cc:1347] Generated new cluster ID: a053a569e8d440369b2fafe0808c6b64
I20260928 08:37:51.650478 6599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:37:51.677765 6599 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:37:51.679157 6599 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:37:51.685856 6599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef: Generated new TSK 0
I20260928 08:37:51.686754 6599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260928 08:37:51.750567 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42403
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:52.076818 6615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:52.077097 6615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:52.077261 6615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:52.085141 6615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:52.085256 6615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:52.085454 6615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:37:52.095356 6615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:52.096680 6615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:52.098037 6615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:52.111281 6620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:52.111233 6623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:52.111320 6621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:52.112097 6615 server_base.cc:1034] running on GCE node
I20260928 08:37:52.112696 6615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:52.113571 6615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:52.114758 6615 hybrid_clock.cc:648] HybridClock initialized: now 1790584672114708 us; error 40 us; skew 500 ppm
I20260928 08:37:52.115062 6615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:52 dist-test-slave-w6c0 krb5kdc[6509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584672, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:52.118033 6615 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260928 08:37:52.119427 6615 webserver.cc:460] Webserver started at http://127.0.97.1:38347/ using document root <none> and password file <none>
I20260928 08:37:52.120066 6615 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:52.120164 6615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:52.120481 6615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:52.123307 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "f1478e5cb49148e4994ac79873f7a0ed"
format_stamp: "Formatted at 2026-09-28 08:37:52 on dist-test-slave-w6c0"
server_key: "6159ebb6c0902a366489f5984bc22424"
server_key_iv: "589e55637db35cf07047122ea1bf428a"
server_key_version: "encryptionkey@0"
I20260928 08:37:52.124054 6615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f1478e5cb49148e4994ac79873f7a0ed"
format_stamp: "Formatted at 2026-09-28 08:37:52 on dist-test-slave-w6c0"
server_key: "6159ebb6c0902a366489f5984bc22424"
server_key_iv: "589e55637db35cf07047122ea1bf428a"
server_key_version: "encryptionkey@0"
I20260928 08:37:52.128818 6615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 08:37:52.132134 6630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:52.133318 6615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:37:52.133473 6615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "f1478e5cb49148e4994ac79873f7a0ed"
format_stamp: "Formatted at 2026-09-28 08:37:52 on dist-test-slave-w6c0"
server_key: "6159ebb6c0902a366489f5984bc22424"
server_key_iv: "589e55637db35cf07047122ea1bf428a"
server_key_version: "encryptionkey@0"
I20260928 08:37:52.133662 6615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:52.189769 6615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:52.194146 6615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:52.194463 6615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:52.195536 6615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:52.196904 6615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:52.196975 6615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:52.197124 6615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:52.197191 6615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:52.230820 6615 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44565
I20260928 08:37:52.230911 6742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44565 every 8 connection(s)
I20260928 08:37:52.232214 6615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:37:52.233557 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 6615
I20260928 08:37:52.233791 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:37:52.234629 388 external_mini_cluster.cc:1428] Setting key 4b73c19ceaba001c4ea3dfb261e80e0e
Sep 28 08:37:52 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584672, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:37:52.256316 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:52.235100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40351 (local address 127.0.97.62:42403)
0928 08:37:52.235586 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:52.235592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:52.238656 (+ 3064us) server_negotiation.cc:407] Connection header received
0928 08:37:52.239654 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:52.239658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:52.239724 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:52.239825 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:52.241944 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:52.242754 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:52.244915 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:52.245659 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:52.245805 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:52.249397 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:52.249406 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:52.249413 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:52.249458 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:52.253057 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:52.253780 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:52.253788 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:52.253792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:52.253933 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:52.254348 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:52.254354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:52.254360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:52.254816 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:52.254986 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:52.255643 (+ 657us) server_negotiation.cc:299] Negotiation successful
0928 08:37:52.255871 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":237,"threads_started":1}
I20260928 08:37:52.257071 6746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:52.235675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.1:40351)
0928 08:37:52.236555 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:52.236656 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:52.239333 (+ 2677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:52.240091 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:52.240105 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:52.240873 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:52.241740 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:52.241766 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:52.242900 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:52.242909 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:52.244657 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:52.244670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:52.245872 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:52.245877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:52.246073 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:52.246876 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:52.246898 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:52.249172 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:52.253210 (+ 4038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:52.253218 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:52.253232 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:52.253657 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:52.254047 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:52.254058 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:52.254061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:52.254235 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:52.254952 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:52.254958 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:52.255274 (+ 316us) client_negotiation.cc:769] Sending connection context
0928 08:37:52.255500 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 08:37:52.255909 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":294,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:37:52.258610 6743 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42403
I20260928 08:37:52.258932 6743 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:52.259626 6743 heartbeater.cc:511] Master 127.0.97.62:42403 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260928 08:37:52.261631 6554 ts_manager.cc:194] Registered new tserver with Master: f1478e5cb49148e4994ac79873f7a0ed (127.0.97.1:44565)
I20260928 08:37:52.263131 6554 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260928 08:37:52.284513 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42403
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:52.615648 6749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:52.615963 6749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:52.616127 6749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:52.623678 6749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:52.623803 6749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:52.624012 6749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:37:52.634280 6749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:52.635689 6749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:52.637204 6749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:52.652222 6755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:52.652187 6757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:52.652575 6754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:52.653733 6749 server_base.cc:1034] running on GCE node
I20260928 08:37:52.654488 6749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:52.655550 6749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:52.656838 6749 hybrid_clock.cc:648] HybridClock initialized: now 1790584672656757 us; error 68 us; skew 500 ppm
I20260928 08:37:52.657207 6749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:52 dist-test-slave-w6c0 krb5kdc[6509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584672, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:52.667167 6749 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260928 08:37:52.668955 6749 webserver.cc:460] Webserver started at http://127.0.97.2:40947/ using document root <none> and password file <none>
I20260928 08:37:52.669718 6749 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:52.669873 6749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:52.670189 6749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:52.673439 6749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "5143bb66a241433383f2648dcb812a64"
format_stamp: "Formatted at 2026-09-28 08:37:52 on dist-test-slave-w6c0"
server_key: "7696d922de121ed599f7353a7b55fdb1"
server_key_iv: "7a27f3941206b035d33ccf8bca3578f3"
server_key_version: "encryptionkey@0"
I20260928 08:37:52.674293 6749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5143bb66a241433383f2648dcb812a64"
format_stamp: "Formatted at 2026-09-28 08:37:52 on dist-test-slave-w6c0"
server_key: "7696d922de121ed599f7353a7b55fdb1"
server_key_iv: "7a27f3941206b035d33ccf8bca3578f3"
server_key_version: "encryptionkey@0"
I20260928 08:37:52.679983 6749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:37:52.684243 6764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:52.685835 6749 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 08:37:52.686030 6749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "5143bb66a241433383f2648dcb812a64"
format_stamp: "Formatted at 2026-09-28 08:37:52 on dist-test-slave-w6c0"
server_key: "7696d922de121ed599f7353a7b55fdb1"
server_key_iv: "7a27f3941206b035d33ccf8bca3578f3"
server_key_version: "encryptionkey@0"
I20260928 08:37:52.686280 6749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:52.707934 6749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:52.713171 6749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:52.713532 6749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:52.714659 6749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:52.716259 6749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:52.716365 6749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:52.716519 6749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:52.716591 6749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:52.751051 6749 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35875
I20260928 08:37:52.751144 6876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35875 every 8 connection(s)
I20260928 08:37:52.752382 6749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:37:52.756381 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 6749
I20260928 08:37:52.756523 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:37:52.757020 388 external_mini_cluster.cc:1428] Setting key 5cbcf308f43834ffb3dd1f10517fd79b
Sep 28 08:37:52 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584672, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:37:52.775388 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:52.755537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57597 (local address 127.0.97.62:42403)
0928 08:37:52.755803 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:52.755808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:52.757743 (+ 1935us) server_negotiation.cc:407] Connection header received
0928 08:37:52.760005 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:52.760010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:52.760097 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:52.760230 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:52.761924 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:52.762726 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:52.764332 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:52.764828 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:52.764966 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:52.769409 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:52.769414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:52.769418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:52.769452 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:52.772222 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:52.772989 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:52.772995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:52.772999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:52.773134 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:52.773567 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:52.773574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:52.773577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:52.773860 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:52.774023 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:52.774808 (+ 785us) server_negotiation.cc:299] Negotiation successful
0928 08:37:52.775041 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260928 08:37:52.776438 6879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:52.756259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.2:57597)
0928 08:37:52.757517 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:52.757580 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:52.759725 (+ 2145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:52.760472 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:52.760481 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:52.761016 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:52.761711 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:52.761735 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:52.762878 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:52.762884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:52.764167 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:52.764180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:52.765185 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:52.765196 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:52.765615 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:52.766354 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:52.766395 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:52.769146 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:52.772384 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:52.772401 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:52.772417 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:52.772857 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:52.773259 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:52.773269 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:52.773271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:52.773455 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:52.773996 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:52.774006 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:52.774378 (+ 372us) client_negotiation.cc:769] Sending connection context
0928 08:37:52.774705 (+ 327us) client_negotiation.cc:240] Negotiation successful
0928 08:37:52.775299 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"thread_start_us":509,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 08:37:52.778045 6877 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42403
I20260928 08:37:52.778437 6877 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:52.779202 6877 heartbeater.cc:511] Master 127.0.97.62:42403 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260928 08:37:52.780790 6554 ts_manager.cc:194] Registered new tserver with Master: 5143bb66a241433383f2648dcb812a64 (127.0.97.2:35875)
I20260928 08:37:52.781602 6554 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260928 08:37:52.806154 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42403
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:53.153079 6882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:53.153359 6882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:53.153451 6882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:53.160733 6882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:53.160852 6882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:53.160967 6882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:37:53.171563 6882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:53.172925 6882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:53.174155 6882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:53.187321 6887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:53.187364 6888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:53.187402 6890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:53.188314 6882 server_base.cc:1034] running on GCE node
I20260928 08:37:53.188922 6882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:53.189975 6882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:53.191181 6882 hybrid_clock.cc:648] HybridClock initialized: now 1790584673191135 us; error 35 us; skew 500 ppm
I20260928 08:37:53.191486 6882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:53 dist-test-slave-w6c0 krb5kdc[6509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584673, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:53.194860 6882 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260928 08:37:53.197007 6882 webserver.cc:460] Webserver started at http://127.0.97.3:42645/ using document root <none> and password file <none>
I20260928 08:37:53.197754 6882 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:53.197896 6882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:53.198199 6882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:53.202596 6882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "3b24f682e35048979dadd681aaf430c6"
format_stamp: "Formatted at 2026-09-28 08:37:53 on dist-test-slave-w6c0"
server_key: "e0e313288c72e23afc4f05ecf26fb223"
server_key_iv: "cee885b63a5ef42636adaad2a80ba83d"
server_key_version: "encryptionkey@0"
I20260928 08:37:53.203742 6882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3b24f682e35048979dadd681aaf430c6"
format_stamp: "Formatted at 2026-09-28 08:37:53 on dist-test-slave-w6c0"
server_key: "e0e313288c72e23afc4f05ecf26fb223"
server_key_iv: "cee885b63a5ef42636adaad2a80ba83d"
server_key_version: "encryptionkey@0"
I20260928 08:37:53.209311 6882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:37:53.213280 6897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:53.215318 6882 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260928 08:37:53.215515 6882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "3b24f682e35048979dadd681aaf430c6"
format_stamp: "Formatted at 2026-09-28 08:37:53 on dist-test-slave-w6c0"
server_key: "e0e313288c72e23afc4f05ecf26fb223"
server_key_iv: "cee885b63a5ef42636adaad2a80ba83d"
server_key_version: "encryptionkey@0"
I20260928 08:37:53.215708 6882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:53.229784 6882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:53.234189 6882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:53.234579 6882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:53.235561 6882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:53.237072 6882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:53.237174 6882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:53.237310 6882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:53.237375 6882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:53.266646 6743 heartbeater.cc:503] Master 127.0.97.62:42403 was elected leader, sending a full tablet report...
I20260928 08:37:53.272450 6882 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40031
I20260928 08:37:53.272552 7009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40031 every 8 connection(s)
I20260928 08:37:53.273720 6882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:37:53.277887 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 6882
I20260928 08:37:53.278060 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:37:53.278479 388 external_mini_cluster.cc:1428] Setting key cac93902a658c810d6652fc6d8459809
Sep 28 08:37:53 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584673, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:37:53.295063 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:53.276346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35795 (local address 127.0.97.62:42403)
0928 08:37:53.276795 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:53.276800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:53.277859 (+ 1059us) server_negotiation.cc:407] Connection header received
0928 08:37:53.279831 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:53.279836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:53.279920 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:53.280055 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:53.281636 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.282424 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.284142 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.284570 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.284714 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:53.288314 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:53.288320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:53.288325 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:53.288367 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:53.291156 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:53.291921 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:53.291925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:53.291927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:53.292128 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:53.292730 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:53.292734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:53.292737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:53.293115 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:53.293299 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:53.294352 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0928 08:37:53.294704 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":236,"threads_started":1}
I20260928 08:37:53.295718 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:53.276897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.3:35795)
0928 08:37:53.277593 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:53.277633 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:53.279542 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:53.280289 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:53.280297 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:53.280793 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:53.281470 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:53.281489 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.282608 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.282619 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:53.283973 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:53.283985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.284810 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.284817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:53.285047 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:53.285766 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:53.285789 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:53.288084 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:53.291315 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:53.291324 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:53.291343 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:53.291767 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:53.292301 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:53.292305 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:53.292308 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:53.292587 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:53.293248 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:53.293254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:53.293723 (+ 469us) client_negotiation.cc:769] Sending connection context
0928 08:37:53.294068 (+ 345us) client_negotiation.cc:240] Negotiation successful
0928 08:37:53.294476 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":259,"threads_started":1}
I20260928 08:37:53.297295 7010 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42403
I20260928 08:37:53.297577 7010 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:53.298257 7010 heartbeater.cc:511] Master 127.0.97.62:42403 requested a full tablet report, sending...
I20260928 08:37:53.299764 6554 ts_manager.cc:194] Registered new tserver with Master: 3b24f682e35048979dadd681aaf430c6 (127.0.97.3:40031)
I20260928 08:37:53.300633 6554 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35795
I20260928 08:37:53.306577 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260928 08:37:53.356529 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38895
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42403,127.0.97.61:38895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:37:53.679910 7016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:53.680188 7016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:53.680342 7016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:53.687631 7016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:37:53.687740 7016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:53.687837 7016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:53.687933 7016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:37:53.688012 7016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:37:53.697794 7016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42403,127.0.97.61:38895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
W20260928 08:37:53.698158 7016 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:42403,127.0.97.61:38895'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 08:37:53.699397 7016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:53.700686 7016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:53.711901 7021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:53.711988 7024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:53.712743 7022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:53.713351 7016 server_base.cc:1034] running on GCE node
I20260928 08:37:53.714063 7016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:53.715451 7016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:53.716692 7016 hybrid_clock.cc:648] HybridClock initialized: now 1790584673716632 us; error 65 us; skew 500 ppm
I20260928 08:37:53.717001 7016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:53 dist-test-slave-w6c0 krb5kdc[6509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584673, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:53.720266 7016 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260928 08:37:53.721654 7016 webserver.cc:460] Webserver started at http://127.0.97.61:33937/ using document root <none> and password file <none>
I20260928 08:37:53.722334 7016 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:53.722461 7016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:53.722786 7016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:53.725682 7016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/data/instance:
uuid: "2590393e2ec44ddfa136fbc2c1f17b79"
format_stamp: "Formatted at 2026-09-28 08:37:53 on dist-test-slave-w6c0"
server_key: "d0045d5f03dbc503dab5611af23cbdd1"
server_key_iv: "b02192632ce9e8a0ab96dc5961e8bffe"
server_key_version: "encryptionkey@0"
I20260928 08:37:53.726436 7016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/wal/instance:
uuid: "2590393e2ec44ddfa136fbc2c1f17b79"
format_stamp: "Formatted at 2026-09-28 08:37:53 on dist-test-slave-w6c0"
server_key: "d0045d5f03dbc503dab5611af23cbdd1"
server_key_iv: "b02192632ce9e8a0ab96dc5961e8bffe"
server_key_version: "encryptionkey@0"
I20260928 08:37:53.731391 7016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:37:53.734576 7031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:53.735809 7016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 08:37:53.735929 7016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/wal
uuid: "2590393e2ec44ddfa136fbc2c1f17b79"
format_stamp: "Formatted at 2026-09-28 08:37:53 on dist-test-slave-w6c0"
server_key: "d0045d5f03dbc503dab5611af23cbdd1"
server_key_iv: "b02192632ce9e8a0ab96dc5961e8bffe"
server_key_version: "encryptionkey@0"
I20260928 08:37:53.736097 7016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:53.757067 7016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:53.760792 7016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:53.761013 7016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:53.779179 7016 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38895
I20260928 08:37:53.779201 7082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38895 every 8 connection(s)
I20260928 08:37:53.780473 7016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/data/info.pb
I20260928 08:37:53.783947 7083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:53.784361 6877 heartbeater.cc:503] Master 127.0.97.62:42403 was elected leader, sending a full tablet report...
I20260928 08:37:53.786042 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 7016
I20260928 08:37:53.793366 7083 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:37:53 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 28 08:37:53 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584673, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:37:53.809250 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:53.788220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57626 (local address 127.0.97.61:38895)
0928 08:37:53.789035 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:53.789045 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:53.789128 (+ 83us) server_negotiation.cc:407] Connection header received
0928 08:37:53.790948 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:53.790973 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:53.791345 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:53.791633 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:53.793406 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.794832 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.796867 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.797526 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.797705 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:53.802215 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:53.802225 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:53.802239 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:53.802284 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:53.805541 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:53.806153 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:53.806163 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:53.806176 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:53.806311 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:53.806741 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:53.806748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:53.806752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:53.807210 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:53.807489 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:53.807906 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 08:37:53.808274 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":389,"threads_started":1}
I20260928 08:37:53.811250 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:53.795969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43242 (local address 127.0.97.62:42403)
0928 08:37:53.796455 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:53.796461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:53.796950 (+ 489us) server_negotiation.cc:407] Connection header received
0928 08:37:53.797146 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:53.797151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:53.797231 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:53.797341 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:53.798798 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.799503 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.801279 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.801933 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.802070 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:53.805897 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:53.805902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:53.805906 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:53.805939 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:53.808607 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:53.809294 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:53.809302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:53.809306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:53.809453 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:53.809830 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:53.809835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:53.809838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:53.810229 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:53.810374 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:53.810860 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 08:37:53.811017 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":222,"threads_started":1}
I20260928 08:37:53.811472 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:53.796011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.0.1:43242)
0928 08:37:53.796813 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:53.796862 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:53.797030 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:53.797393 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:53.797400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:53.798262 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:53.798638 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:53.798655 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.799662 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.799670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:53.801107 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:53.801118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.802168 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.802179 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:53.802401 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:53.803063 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:53.803086 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:53.805707 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:53.808740 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:53.808750 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:53.808756 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:53.809168 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:53.809582 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:53.809586 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:53.809588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:53.809713 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:53.810350 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:53.810355 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:53.810541 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 08:37:53.810767 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 08:37:53.811087 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":410,"threads_started":1}
I20260928 08:37:53.812927 7083 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:37:53 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584673, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260928 08:37:53.824434 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:53.813498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38895 (local address 127.0.0.1:57636)
0928 08:37:53.813698 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:53.813716 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:53.813796 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:53.814017 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:53.814020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:53.814306 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:53.814421 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:53.814428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.815345 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.815348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:53.816181 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:53.816191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.816857 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.816861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:53.816995 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:53.817448 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:53.817463 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:53.819373 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:53.822133 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:53.822138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:53.822142 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:53.822487 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:53.822879 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:53.822885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:53.822889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:53.823012 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:53.823511 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:53.823516 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:53.823661 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 08:37:53.823790 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 08:37:53.824053 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 08:37:53.824496 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:53.813672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.0.97.61:38895)
0928 08:37:53.813815 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:53.813819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:53.813832 (+ 13us) server_negotiation.cc:407] Connection header received
0928 08:37:53.813865 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:53.813868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:53.813916 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:53.813986 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:53.814591 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.815244 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.816326 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.816726 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.816833 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:53.819546 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:53.819550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:53.819553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:53.819579 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:53.821973 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:53.822656 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:53.822664 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:53.822668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:53.822777 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:53.823115 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:53.823120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:53.823124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:53.823376 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:53.823532 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:53.823856 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 08:37:53.824190 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":49}
I20260928 08:37:53.829643 7083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: Bootstrap starting.
I20260928 08:37:53.832666 7083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:53.833573 7083 log.cc:824] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:53.836061 7083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: No bootstrap required, opened a new log
I20260928 08:37:53.843137 7083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } }
I20260928 08:37:53.843416 7083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:53.843499 7083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2590393e2ec44ddfa136fbc2c1f17b79, State: Initialized, Role: FOLLOWER
I20260928 08:37:53.843979 7083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } }
I20260928 08:37:53.844704 7090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } } }
I20260928 08:37:53.844970 7090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:37:53.845677 7083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 08:37:53.853668 7101 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:37:53.853801 7101 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:37:53.856295 7016 master_runner.cc:428] Detected that this master 2590393e2ec44ddfa136fbc2c1f17b79 is joining an existing cluster
I20260928 08:37:53.856406 7016 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:38895
I20260928 08:37:53.876081 7107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:53.860091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.0.1:43250)
0928 08:37:53.860670 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:53.860686 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:53.860837 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:53.861243 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:53.861248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:53.861808 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:53.862099 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:53.862107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.863552 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.863586 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:53.865341 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:53.865357 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.866676 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.866685 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:53.866896 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:53.867612 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:53.867640 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:53.868974 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:53.872929 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:53.872937 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:53.872943 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:53.873478 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:53.873931 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:53.873938 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:53.873944 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:53.874128 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:53.874698 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:53.874703 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:53.874929 (+ 226us) client_negotiation.cc:769] Sending connection context
0928 08:37:53.875103 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 08:37:53.875471 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":321,"threads_started":1}
I20260928 08:37:53.877132 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:53.860038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43250 (local address 127.0.97.62:42403)
0928 08:37:53.860204 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:53.860208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:53.860766 (+ 558us) server_negotiation.cc:407] Connection header received
0928 08:37:53.860960 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:53.860967 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:53.861064 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:53.861171 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:53.862238 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.863350 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.865515 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.866268 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.866402 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:53.869180 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:53.869189 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:53.869197 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:53.869237 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:53.872775 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:53.873617 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:53.873627 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:53.873631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:53.873786 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:53.874244 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:53.874251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:53.874256 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:53.874567 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:53.874732 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:53.876469 (+ 1737us) server_negotiation.cc:299] Negotiation successful
0928 08:37:53.876714 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 08:37:53.880154 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:53.861111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57650 (local address 127.0.97.61:38895)
0928 08:37:53.861326 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:53.861336 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:53.862794 (+ 1458us) server_negotiation.cc:407] Connection header received
0928 08:37:53.862920 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:53.862924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:53.862987 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:53.863093 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:53.864507 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.865629 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.867898 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.868595 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.868686 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:53.871585 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:53.871591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:53.871596 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:53.871629 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:53.875285 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:53.876112 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:53.876117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:53.876121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:53.876258 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:53.877000 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:53.877004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:53.877007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:53.877496 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:53.877655 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:53.878143 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 08:37:53.879772 (+ 1629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 08:37:53.881108 7108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:53.860847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38895 (local address 127.0.0.1:57650)
0928 08:37:53.862686 (+ 1839us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:53.862704 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:53.862818 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:53.863229 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:53.863235 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:53.863993 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:53.864355 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:53.864368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.865837 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.865842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:53.867746 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:53.867765 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.868985 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.868991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:53.869161 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:53.869768 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:53.869789 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:53.871416 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:53.875427 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:53.875432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:53.875436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:53.875966 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:53.876717 (+ 751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:53.876721 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:53.876725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:53.876880 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:53.877730 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:53.877734 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:53.877891 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 08:37:53.878580 (+ 689us) client_negotiation.cc:240] Negotiation successful
0928 08:37:53.880651 (+ 2071us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1707,"mutex_wait_us":1759,"thread_start_us":1501,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 28 08:37:53 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584671, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260928 08:37:53.899211 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:53.883651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.0.97.61:38895)
0928 08:37:53.883816 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:53.883822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:53.884221 (+ 399us) server_negotiation.cc:407] Connection header received
0928 08:37:53.884556 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:53.884562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:53.884652 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:53.884762 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:53.886461 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.887689 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.889546 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.890222 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.890323 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:53.893799 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:53.893804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:53.893809 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:53.893847 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:53.896472 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:53.897161 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:53.897166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:53.897174 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:53.897293 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:53.897658 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:53.897669 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:53.897674 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:53.897935 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:53.898090 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:53.898580 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 08:37:53.898812 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 08:37:53.899261 7110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:53.883265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38895 (local address 127.0.0.1:57664)
0928 08:37:53.884098 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:53.884139 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:53.884340 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:53.885032 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:53.885043 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:53.885941 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:53.886274 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:53.886290 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.887818 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.887824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:53.889399 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:53.889414 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.890386 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.890393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:53.890613 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:53.891321 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:53.891342 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:53.893620 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:53.896600 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:53.896613 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:53.896620 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:53.897034 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:53.897399 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:53.897407 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:53.897412 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:53.897544 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:53.898065 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:53.898073 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:53.898265 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 08:37:53.898453 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 08:37:53.898813 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":435,"threads_started":1}
I20260928 08:37:53.901795 6554 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:38895: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 38895
}
attrs {
promote: true
}
}
dest_uuid: "899270827952449d802397379a7ed6ef"
cas_config_opid_index: -1
I20260928 08:37:53.903054 6554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: true } }
I20260928 08:37:53.911456 7058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:53.911171 7114 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2590393e2ec44ddfa136fbc2c1f17b79 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: true } } }
I20260928 08:37:53.915036 7058 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 1 FOLLOWER]: Refusing update from remote peer 899270827952449d802397379a7ed6ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 08:37:53.916034 7113 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:53.917142 7114 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:38895
I20260928 08:37:53.917791 7016 master.cc:537] Master@127.0.97.61:38895 shutting down...
W20260928 08:37:53.919478 6538 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260928 08:37:53.920915 7115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "899270827952449d802397379a7ed6ef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: true } } }
I20260928 08:37:53.921255 7115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: This master's current role is: LEADER
W20260928 08:37:53.921743 6538 consensus_peers.cc:487] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef -> Peer 2590393e2ec44ddfa136fbc2c1f17b79 (127.0.97.61:38895): Couldn't send request to peer 2590393e2ec44ddfa136fbc2c1f17b79. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260928 08:37:53.926613 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40117
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42403,127.0.97.61:38895,127.0.97.60:40117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:37:53.931463 7016 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:53.931946 7016 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:53.932163 7016 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: stopping tablet replica
I20260928 08:37:53.939013 7016 master.cc:552] Master@127.0.97.61:38895 shutdown complete.
I20260928 08:37:53.939110 7016 master_runner.cc:305] Clearing existing system tablet
I20260928 08:37:53.939572 7016 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:53.939670 7016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:53.942891 7016 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 08:37:53.944654 7121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:53.945080 7016 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:53.945454 7016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 08:37:53.945544 7016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/wal
uuid: "2590393e2ec44ddfa136fbc2c1f17b79"
format_stamp: "Formatted at 2026-09-28 08:37:53 on dist-test-slave-w6c0"
server_key: "d0045d5f03dbc503dab5611af23cbdd1"
server_key_iv: "b02192632ce9e8a0ab96dc5961e8bffe"
server_key_version: "encryptionkey@0"
I20260928 08:37:53.952147 7016 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:53.959441 7016 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 08:37:53.959626 7016 log.cc:1196] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 08:37:53.960090 7016 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: Deleting consensus metadata
I20260928 08:37:53.960347 7016 master_runner.cc:315] Copying system tablet from 127.0.97.62:42403
I20260928 08:37:53.962983 7016 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:42403
I20260928 08:37:53.985280 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:53.963911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43258 (local address 127.0.97.62:42403)
0928 08:37:53.964105 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:53.964112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:53.971168 (+ 7056us) server_negotiation.cc:407] Connection header received
0928 08:37:53.971364 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:53.971370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:53.971471 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:53.971589 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:53.972701 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.973837 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.975645 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.976327 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.976465 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:53.978730 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:53.978738 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:53.978743 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:53.978781 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:53.982222 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:53.983097 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:53.983103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:53.983107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:53.983239 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:53.983683 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:53.983689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:53.983692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:53.984030 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:53.984195 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:53.984666 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 08:37:53.984900 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 08:37:53.985309 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:53.963769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.0.1:43258)
0928 08:37:53.970984 (+ 7215us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:53.971014 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:53.971213 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:53.971637 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:53.971642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:53.972216 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:53.972527 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:53.972543 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.973996 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.974002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:53.975494 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:53.975509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:53.976502 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:53.976507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:53.976663 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:53.977445 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:53.977467 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:53.978467 (+ 1000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:53.982373 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:53.982379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:53.982386 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:53.982946 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:53.983361 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:53.983367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:53.983370 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:53.983558 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:53.984175 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:53.984180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:53.984363 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 08:37:53.984535 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 08:37:53.984865 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7057,"thread_start_us":6892,"threads_started":1}
I20260928 08:37:53.986491 6574 tablet_copy_service.cc:140] P 899270827952449d802397379a7ed6ef: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2590393e2ec44ddfa136fbc2c1f17b79 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:43258)
I20260928 08:37:53.986817 6574 tablet_copy_service.cc:161] P 899270827952449d802397379a7ed6ef: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2590393e2ec44ddfa136fbc2c1f17b79 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:43258: session id = 2590393e2ec44ddfa136fbc2c1f17b79-00000000000000000000000000000000
I20260928 08:37:53.991533 6574 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 08:37:53.993855 7016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 08:37:54.000710 7016 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: tablet copy: Starting download of 0 data blocks...
I20260928 08:37:54.001040 7016 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: tablet copy: Starting download of 1 WAL segments...
I20260928 08:37:54.003826 7016 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 08:37:54.008168 6574 tablet_copy_service.cc:342] P 899270827952449d802397379a7ed6ef: Request end of tablet copy session 2590393e2ec44ddfa136fbc2c1f17b79-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:43258
I20260928 08:37:54.008378 6574 tablet_copy_service.cc:433] P 899270827952449d802397379a7ed6ef: ending tablet copy session 2590393e2ec44ddfa136fbc2c1f17b79-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2590393e2ec44ddfa136fbc2c1f17b79
I20260928 08:37:54.020576 7016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:54.023422 7128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:54.027770 7129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:54.036336 7016 server_base.cc:1034] running on GCE node
W20260928 08:37:54.036698 7131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:54.037283 7016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:54.038087 7016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:54.039374 7016 hybrid_clock.cc:648] HybridClock initialized: now 1790584674039303 us; error 37 us; skew 500 ppm
I20260928 08:37:54.039690 7016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:54 dist-test-slave-w6c0 krb5kdc[6509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584674, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:54.042274 7016 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260928 08:37:54.043715 7016 webserver.cc:460] Webserver started at http://127.0.97.61:37297/ using document root <none> and password file <none>
I20260928 08:37:54.044152 7016 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:54.044294 7016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:54.047686 7016 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 08:37:54.050190 7138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:54.050668 7016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:37:54.050879 7016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/wal
uuid: "2590393e2ec44ddfa136fbc2c1f17b79"
format_stamp: "Formatted at 2026-09-28 08:37:53 on dist-test-slave-w6c0"
server_key: "d0045d5f03dbc503dab5611af23cbdd1"
server_key_iv: "b02192632ce9e8a0ab96dc5961e8bffe"
server_key_version: "encryptionkey@0"
I20260928 08:37:54.051121 7016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:54.068814 7016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:54.073557 7016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:54.098496 7016 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38895
I20260928 08:37:54.098567 7189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38895 every 8 connection(s)
I20260928 08:37:54.099670 7016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-1/data/info.pb
I20260928 08:37:54.105571 7190 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:37:54.109970 7190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: Bootstrap starting.
I20260928 08:37:54.126992 7190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:37:54.127661 7190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: Bootstrap complete.
I20260928 08:37:54.128746 7190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: true } }
I20260928 08:37:54.129110 7190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2590393e2ec44ddfa136fbc2c1f17b79, State: Initialized, Role: LEARNER
I20260928 08:37:54.129380 7190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: true } }
I20260928 08:37:54.130299 7193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: true } } }
I20260928 08:37:54.130594 7190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:37:54.130687 7193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [sys.catalog]: This master's current role is: LEARNER
I20260928 08:37:54.151338 7204 catalog_manager.cc:1259] Loaded cluster ID: a053a569e8d440369b2fafe0808c6b64
I20260928 08:37:54.151453 7204 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: loading cluster ID for follower catalog manager: success
I20260928 08:37:54.155076 7204 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: acquiring CA information for follower catalog manager: success
I20260928 08:37:54.157398 7204 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:37:54.303345 7010 heartbeater.cc:503] Master 127.0.97.62:42403 was elected leader, sending a full tablet report...
W20260928 08:37:54.331964 7119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:54.332279 7119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:54.332435 7119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:54.340111 7119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:37:54.340245 7119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:54.340353 7119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:54.340462 7119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:37:54.340539 7119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:37:54.350831 7119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42403,127.0.97.61:38895,127.0.97.60:40117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:54.352335 7119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:54.353734 7119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:54.365741 7209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:54.365706 7210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:54.366884 7119 server_base.cc:1034] running on GCE node
W20260928 08:37:54.365681 7212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:54.368104 7119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:54.369393 7119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:54.370599 7119 hybrid_clock.cc:648] HybridClock initialized: now 1790584674370547 us; error 35 us; skew 500 ppm
I20260928 08:37:54.370929 7119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:54 dist-test-slave-w6c0 krb5kdc[6509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584674, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:54.373703 7110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.367413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38895 (local address 127.0.0.1:58322)
0928 08:37:54.367620 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:54.367637 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:54.367740 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:54.368482 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:54.368487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:54.368514 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:54.368725 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:54.368734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.370339 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.370345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:54.371749 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:54.371762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.372892 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.372898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:54.373129 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.373188 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 08:37:54.373302 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 08:37:54.373434 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 08:37:54.373769 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.367581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58322 (local address 127.0.97.61:38895)
0928 08:37:54.368114 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:54.368118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:54.368136 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:37:54.368225 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:54.368229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:54.368329 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:54.368450 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:54.368897 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.370196 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.371895 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.372729 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.372876 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.373098 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:54.373326 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 08:37:54.373457 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":291,"threads_started":1}
I20260928 08:37:54.374244 7119 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260928 08:37:54.374925 7165 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260928 08:37:54.375500 7193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 899270827952449d802397379a7ed6ef. Latest consensus state: current_term: 1 leader_uuid: "899270827952449d802397379a7ed6ef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: true } } }
I20260928 08:37:54.375746 7193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [sys.catalog]: This master's current role is: LEARNER
I20260928 08:37:54.376006 7119 webserver.cc:460] Webserver started at http://127.0.97.60:44489/ using document root <none> and password file <none>
I20260928 08:37:54.376686 7119 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:54.377228 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:54.377547 7119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:37:54.380769 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/data/instance:
uuid: "0ee6753344fc4449bad50ab34692b2ee"
format_stamp: "Formatted at 2026-09-28 08:37:54 on dist-test-slave-w6c0"
server_key: "f076bf4c1da216c89ee717b9fce100ec"
server_key_iv: "6fb26831860e5008fee2e4f7c82a7c03"
server_key_version: "encryptionkey@0"
I20260928 08:37:54.381536 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/wal/instance:
uuid: "0ee6753344fc4449bad50ab34692b2ee"
format_stamp: "Formatted at 2026-09-28 08:37:54 on dist-test-slave-w6c0"
server_key: "f076bf4c1da216c89ee717b9fce100ec"
server_key_iv: "6fb26831860e5008fee2e4f7c82a7c03"
server_key_version: "encryptionkey@0"
I20260928 08:37:54.386891 7119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 08:37:54.390278 7220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:54.391644 7119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:37:54.391816 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/wal
uuid: "0ee6753344fc4449bad50ab34692b2ee"
format_stamp: "Formatted at 2026-09-28 08:37:54 on dist-test-slave-w6c0"
server_key: "f076bf4c1da216c89ee717b9fce100ec"
server_key_iv: "6fb26831860e5008fee2e4f7c82a7c03"
server_key_version: "encryptionkey@0"
I20260928 08:37:54.392009 7119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:54.413246 7119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:54.417413 7119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:54.417694 7119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:54.436931 7119 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40117
I20260928 08:37:54.436967 7271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40117 every 8 connection(s)
I20260928 08:37:54.438359 7119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/data/info.pb
I20260928 08:37:54.442442 7272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:54.443647 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 7119
I20260928 08:37:54.450956 7272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:37:54 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 28 08:37:54 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584674, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:37:54.468405 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.445748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.0.97.60:40117)
0928 08:37:54.446708 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:54.446721 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:54.446762 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:37:54.448598 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:54.448619 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:54.449177 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:54.449624 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:54.451232 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.452764 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.454680 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.455721 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.455982 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.460469 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:54.460486 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:54.460503 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:54.460540 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:54.463978 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.464791 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.464798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.464804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.464967 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.465350 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.465354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.465356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.465827 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:54.466072 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:54.466596 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 08:37:54.467057 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":661,"thread_start_us":317,"threads_started":1}
I20260928 08:37:54.468554 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.453427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57268 (local address 127.0.97.62:42403)
0928 08:37:54.453675 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:54.453684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:54.454242 (+ 558us) server_negotiation.cc:407] Connection header received
0928 08:37:54.454442 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:54.454446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:54.454561 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:54.454673 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:54.456244 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.457111 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.458400 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.458888 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.459026 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.462872 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:54.462876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:54.462880 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:54.462907 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:54.465709 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.466581 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.466587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.466591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.466743 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.467185 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.467189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.467191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.467456 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:54.467558 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:54.468105 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 08:37:54.468263 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 08:37:54.468865 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.453354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.0.1:57268)
0928 08:37:54.454099 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:54.454145 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:54.454318 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:54.454728 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:54.454735 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:54.455771 (+ 1036us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:54.456100 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:54.456109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.457249 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.457256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:54.458259 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:54.458270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.459106 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.459113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:54.459255 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.459890 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:54.459917 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:54.462693 (+ 2776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:54.465830 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:54.465842 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:54.465849 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:54.466401 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:54.466892 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:54.466897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:54.466902 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:54.467068 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:54.467603 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:54.467611 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:54.467801 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 08:37:54.468031 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 08:37:54.468346 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":56,"thread_start_us":362,"threads_started":1}
I20260928 08:37:54.470541 7272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:37:54 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584674, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260928 08:37:54.482821 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.471160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38895 (local address 127.0.0.1:58332)
0928 08:37:54.471774 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:54.471789 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:54.471888 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:54.472283 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:54.472287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:54.472610 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:54.472732 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:54.472740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.473696 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.473704 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:54.474668 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:54.474679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.475432 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.475437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:54.475634 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.476112 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:54.476128 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:54.477989 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:54.480762 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:54.480765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:54.480768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:54.481104 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:54.481458 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:54.481461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:54.481464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:54.481600 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:54.482104 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:54.482108 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:54.482216 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 08:37:54.482331 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 08:37:54.482612 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260928 08:37:54.482882 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.471818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58332 (local address 127.0.97.61:38895)
0928 08:37:54.472017 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:54.472024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:54.472041 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:37:54.472103 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:54.472106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:54.472163 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:54.472266 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:54.472869 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.473573 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.474806 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.475244 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.475362 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.478161 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:54.478165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:54.478168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:54.478197 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:54.480636 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.481198 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.481202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.481204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.481315 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.481718 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.481722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.481724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.481994 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:54.482106 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:54.482434 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 08:37:54.482666 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 08:37:54.483774 7272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:37:54 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584674, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260928 08:37:54.496102 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.484391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40117 (local address 127.0.0.1:57022)
0928 08:37:54.484558 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:54.484577 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:54.484662 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:54.485037 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:54.485040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:54.485377 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:54.485523 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:54.485531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.486434 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.486437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:54.487304 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:54.487314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.488033 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.488037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:54.488181 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.488718 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:54.488734 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:54.490783 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:54.493914 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:54.493918 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:54.493920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:54.494245 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:54.494662 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:54.494665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:54.494667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:54.494801 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:54.495295 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:54.495298 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:54.495402 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 08:37:54.495515 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 08:37:54.495724 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 08:37:54.496280 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.484649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57022 (local address 127.0.97.60:40117)
0928 08:37:54.484808 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:54.484812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:54.484827 (+ 15us) server_negotiation.cc:407] Connection header received
0928 08:37:54.484862 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:54.484865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:54.484912 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:54.485000 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:54.485640 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.486285 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.487473 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.487892 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.487997 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.490988 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:54.490997 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:54.491002 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:54.491042 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:54.493794 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.494357 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.494361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.494363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.494486 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.494912 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.494917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.494919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.495189 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:54.495308 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:54.495597 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 08:37:54.495830 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":71}
I20260928 08:37:54.500962 7272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: Bootstrap starting.
I20260928 08:37:54.504004 7272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:54.504974 7272 log.cc:824] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:54.507373 7272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: No bootstrap required, opened a new log
I20260928 08:37:54.514573 7272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } }
I20260928 08:37:54.514873 7272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:54.514957 7272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee6753344fc4449bad50ab34692b2ee, State: Initialized, Role: FOLLOWER
I20260928 08:37:54.515499 7272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } }
I20260928 08:37:54.516234 7278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } } }
I20260928 08:37:54.516525 7278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:37:54.517210 7272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:37:54.523356 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 6615
W20260928 08:37:54.527920 7289 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:37:54.528054 7289 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:37:54.530997 7119 master_runner.cc:428] Detected that this master 0ee6753344fc4449bad50ab34692b2ee is joining an existing cluster
I20260928 08:37:54.531108 7119 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:40117
Sep 28 08:37:54 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260928 08:37:54.549242 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.524741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37382 (local address 127.0.97.1:44565)
0928 08:37:54.525332 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:54.525340 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:54.525366 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:37:54.525430 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:54.525438 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:54.525688 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:54.528136 (+ 2448us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:54.528392 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.529804 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.531779 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.532461 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.532615 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.538375 (+ 5760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:54.538393 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:54.538415 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:54.538472 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:54.543981 (+ 5509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.545041 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.545051 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.545055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.545241 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.545845 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.545853 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.545858 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.546368 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:54.548102 (+ 1734us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:54.548316 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 08:37:54.548662 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":2002,"threads_started":1}
I20260928 08:37:54.551388 6738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:54.582099 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.537595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.0.97.60:40117)
0928 08:37:54.537752 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:54.537757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:54.546828 (+ 9071us) server_negotiation.cc:407] Connection header received
0928 08:37:54.546929 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:54.546934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:54.547015 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:54.547114 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:54.552179 (+ 5065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.553323 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.558764 (+ 5441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.559512 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.559609 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.568224 (+ 8615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:54.568230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:54.568235 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:54.568268 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:54.576417 (+ 8149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.578060 (+ 1643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.578065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.578069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.578239 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.578862 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.578868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.578871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.579230 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:54.579662 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:54.581489 (+ 1827us) server_negotiation.cc:299] Negotiation successful
0928 08:37:54.581709 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 08:37:54.583213 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.537077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40117 (local address 127.0.0.1:57036)
0928 08:37:54.546690 (+ 9613us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:54.546711 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:54.546826 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:54.547272 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:54.547277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:54.551689 (+ 4412us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:54.552034 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:54.552048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.553549 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.553554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:54.558616 (+ 5062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:54.558630 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.559924 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.559930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:54.560107 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.563667 (+ 3560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:54.563697 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:54.564916 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:54.576638 (+ 11722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:54.576643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:54.576647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:54.577174 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:54.578465 (+ 1291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:54.578469 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:54.578473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:54.578660 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:54.579797 (+ 1137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:54.579802 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:54.579969 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 08:37:54.582420 (+ 2451us) client_negotiation.cc:240] Negotiation successful
0928 08:37:54.582747 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9473,"thread_start_us":9253,"threads_started":1}
I20260928 08:37:54.587158 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.539906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58346 (local address 127.0.97.61:38895)
0928 08:37:54.540136 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:54.540145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:54.567955 (+ 27810us) server_negotiation.cc:407] Connection header received
0928 08:37:54.568094 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:54.568102 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:54.568206 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:54.568327 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:54.570342 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.571831 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.574544 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.575441 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.575550 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.578779 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:54.578788 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:54.578796 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:54.578848 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:54.583325 (+ 4477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.584312 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.584321 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.584327 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.584510 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.585072 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.585080 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.585085 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.585506 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:54.585643 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:54.586384 (+ 741us) server_negotiation.cc:299] Negotiation successful
0928 08:37:54.586697 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:37:54.587553 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.537522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57270 (local address 127.0.97.62:42403)
0928 08:37:54.537726 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:54.537731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:54.537750 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:37:54.537806 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:54.537810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:54.537883 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:54.537978 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:54.554373 (+ 16395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.555614 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.560761 (+ 5147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.561434 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.561538 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.569363 (+ 7825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:54.569370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:54.569375 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:54.569416 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:54.574053 (+ 4637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.580229 (+ 6176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.580234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.580237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.580806 (+ 569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.585675 (+ 4869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.585682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.585686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.586055 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:54.586214 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:54.586920 (+ 706us) server_negotiation.cc:299] Negotiation successful
0928 08:37:54.587179 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:37:54.588511 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.538153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38895 (local address 127.0.0.1:58346)
0928 08:37:54.567819 (+ 29666us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:54.567843 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:54.567966 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:54.568562 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:54.568570 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:54.569687 (+ 1117us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:54.570135 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:54.570148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.572076 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.572084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:54.574309 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:54.574329 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.575904 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.575912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:54.576169 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.576997 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:54.577029 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:54.578519 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:54.583507 (+ 4988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:54.583514 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:54.583520 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:54.584146 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:54.584677 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:54.584684 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:54.584689 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:54.584917 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:54.585753 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:54.585760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:54.586017 (+ 257us) client_negotiation.cc:769] Sending connection context
0928 08:37:54.587412 (+ 1395us) client_negotiation.cc:240] Negotiation successful
0928 08:37:54.587783 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29519,"thread_start_us":29374,"threads_started":1}
I20260928 08:37:54.588630 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.536009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.0.1:57270)
0928 08:37:54.537211 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:54.537232 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:54.537355 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:54.538164 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:54.538169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:54.550540 (+ 12371us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:54.550931 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:54.550944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.555818 (+ 4874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.555823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:54.557556 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:54.557571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.561913 (+ 4342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.561920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:54.562083 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.565634 (+ 3551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:54.565655 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:54.567179 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:54.577378 (+ 10199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:54.577385 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:54.577393 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:54.577838 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:54.581010 (+ 3172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:54.581015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:54.581019 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:54.581229 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:54.586335 (+ 5106us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:54.586340 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:54.586598 (+ 258us) client_negotiation.cc:769] Sending connection context
0928 08:37:54.587809 (+ 1211us) client_negotiation.cc:240] Negotiation successful
0928 08:37:54.588146 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":1762,"threads_started":1}
Sep 28 08:37:54 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584671, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260928 08:37:54.610680 7110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.594349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40117 (local address 127.0.0.1:57042)
0928 08:37:54.594611 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:54.594631 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:54.594741 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:54.595497 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:54.595503 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:54.595973 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:54.596199 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:54.596209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.597588 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.597593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:54.599104 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:54.599117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.600294 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.600300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:54.600475 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.601070 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:54.601091 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:54.603789 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:54.608089 (+ 4300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:54.608094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:54.608099 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:54.608567 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:54.608959 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:54.608963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:54.608966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:54.609116 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:54.609724 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:54.609729 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:54.609877 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 08:37:54.609993 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 08:37:54.610269 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 08:37:54.611578 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.594952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57042 (local address 127.0.97.60:40117)
0928 08:37:54.595112 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:54.595118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:54.595149 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:37:54.595237 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:54.595243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:54.595356 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:54.595463 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:54.596319 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.597428 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.599271 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.599963 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.600095 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.603985 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:54.603992 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:54.603997 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:54.604030 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:54.607933 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.608697 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.608702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.608707 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.608836 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.609231 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.609236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.609240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.609548 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:54.609672 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:54.611049 (+ 1377us) server_negotiation.cc:299] Negotiation successful
0928 08:37:54.611262 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:37:54.612995 6554 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:40117: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 40117
}
attrs {
promote: true
}
}
dest_uuid: "899270827952449d802397379a7ed6ef"
cas_config_opid_index: 5
I20260928 08:37:54.613830 6554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: true } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: true } }
I20260928 08:37:54.615689 7114 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0ee6753344fc4449bad50ab34692b2ee (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: true } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: true } } }
I20260928 08:37:54.619465 7165 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 1 LEARNER]: Refusing update from remote peer 899270827952449d802397379a7ed6ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 08:37:54.621320 7247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:54.628535 7114 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:40117
I20260928 08:37:54.629588 7119 master.cc:537] Master@127.0.97.60:40117 shutting down...
I20260928 08:37:54.631721 7247 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [term 1 FOLLOWER]: Refusing update from remote peer 899270827952449d802397379a7ed6ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 08:37:54.642894 7302 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 08:37:54.649559 7302 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:54.650723 7300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "899270827952449d802397379a7ed6ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: true } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: true } } }
I20260928 08:37:54.651600 7300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:54.657817 7165 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0ee6753344fc4449bad50ab34692b2ee (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: true } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: true } } }
I20260928 08:37:54.660805 7193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "899270827952449d802397379a7ed6ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: true } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: true } } }
I20260928 08:37:54.661087 7193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [sys.catalog]: This master's current role is: LEARNER
I20260928 08:37:54.661815 7302 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef: attempting to promote NON_VOTER 2590393e2ec44ddfa136fbc2c1f17b79 to VOTER
I20260928 08:37:54.662462 7302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: true } }
I20260928 08:37:54.665000 7165 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 1 LEARNER]: Refusing update from remote peer 899270827952449d802397379a7ed6ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 08:37:54.665673 7302 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 08:37:54.666062 7119 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:54.666456 7119 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:54.666734 7119 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: stopping tablet replica
W20260928 08:37:54.670858 6540 consensus_peers.cc:487] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef -> Peer 0ee6753344fc4449bad50ab34692b2ee (127.0.97.60:40117): Couldn't send request to peer 0ee6753344fc4449bad50ab34692b2ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 08:37:54.673696 7300 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2590393e2ec44ddfa136fbc2c1f17b79 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: true } } }
I20260928 08:37:54.674551 7119 master.cc:552] Master@127.0.97.60:40117 shutdown complete.
I20260928 08:37:54.674636 7119 master_runner.cc:305] Clearing existing system tablet
I20260928 08:37:54.675007 7119 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:54.674549 7164 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2590393e2ec44ddfa136fbc2c1f17b79 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: true } } }
I20260928 08:37:54.675107 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:54.678128 7302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "899270827952449d802397379a7ed6ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: true } } }
I20260928 08:37:54.678489 7302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:54.678701 7193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "899270827952449d802397379a7ed6ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: true } } }
I20260928 08:37:54.678956 7193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:37:54.678850 7302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "899270827952449d802397379a7ed6ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: true } } }
I20260928 08:37:54.679221 7302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:54.680229 7119 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 08:37:54.681703 7307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:54.682014 7119 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:54.682294 7119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:37:54.682391 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/wal
uuid: "0ee6753344fc4449bad50ab34692b2ee"
format_stamp: "Formatted at 2026-09-28 08:37:54 on dist-test-slave-w6c0"
server_key: "f076bf4c1da216c89ee717b9fce100ec"
server_key_iv: "6fb26831860e5008fee2e4f7c82a7c03"
server_key_version: "encryptionkey@0"
I20260928 08:37:54.686164 7119 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:54.691089 7119 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 08:37:54.691249 7119 log.cc:1196] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 08:37:54.691665 7119 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: Deleting consensus metadata
I20260928 08:37:54.691900 7119 master_runner.cc:315] Copying system tablet from 127.0.97.62:42403
I20260928 08:37:54.694233 7119 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:42403
I20260928 08:37:54.709981 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.695003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57280 (local address 127.0.97.62:42403)
0928 08:37:54.695246 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:54.695253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:54.699107 (+ 3854us) server_negotiation.cc:407] Connection header received
0928 08:37:54.699324 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:54.699330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:54.699398 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:54.699506 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:54.700322 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.701053 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.702378 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.702872 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.703010 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.704806 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:54.704812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:54.704817 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:54.704856 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:54.707465 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.708092 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.708100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.708104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.708230 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:54.708598 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:54.708606 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:54.708609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:54.708867 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:54.709019 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:54.709397 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 08:37:54.709617 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 08:37:54.710047 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:54.694955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.0.1:57280)
0928 08:37:54.698996 (+ 4041us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:54.699013 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:54.699180 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:54.699535 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:54.699539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:54.699977 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:54.700180 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:54.700188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.701190 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.701196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:54.702239 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:54.702251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:54.703025 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:54.703031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:54.703196 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:54.703893 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:54.703920 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:54.704613 (+ 693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:54.707596 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:54.707602 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:54.707606 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:54.707973 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:54.708339 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:54.708344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:54.708348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:54.708492 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:54.708986 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:54.708991 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:54.709157 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 08:37:54.709297 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 08:37:54.709605 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3886,"thread_start_us":3786,"threads_started":1}
I20260928 08:37:54.710922 6574 tablet_copy_service.cc:140] P 899270827952449d802397379a7ed6ef: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0ee6753344fc4449bad50ab34692b2ee ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:57280)
I20260928 08:37:54.711088 6574 tablet_copy_service.cc:161] P 899270827952449d802397379a7ed6ef: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0ee6753344fc4449bad50ab34692b2ee at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:57280: session id = 0ee6753344fc4449bad50ab34692b2ee-00000000000000000000000000000000
I20260928 08:37:54.713274 6574 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 08:37:54.714948 7119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 08:37:54.720474 7119 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: tablet copy: Starting download of 0 data blocks...
I20260928 08:37:54.720746 7119 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: tablet copy: Starting download of 1 WAL segments...
I20260928 08:37:54.722908 7119 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 08:37:54.725862 6574 tablet_copy_service.cc:342] P 899270827952449d802397379a7ed6ef: Request end of tablet copy session 0ee6753344fc4449bad50ab34692b2ee-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:57280
I20260928 08:37:54.726001 6574 tablet_copy_service.cc:433] P 899270827952449d802397379a7ed6ef: ending tablet copy session 0ee6753344fc4449bad50ab34692b2ee-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0ee6753344fc4449bad50ab34692b2ee
I20260928 08:37:54.735867 7119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:54.738566 7314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:54.744006 7315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:54.753489 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 6615
W20260928 08:37:54.768147 7317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:54.768595 7119 server_base.cc:1034] running on GCE node
I20260928 08:37:54.768946 7119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:54.769640 7119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:54.770839 7119 hybrid_clock.cc:648] HybridClock initialized: now 1790584674770778 us; error 49 us; skew 500 ppm
I20260928 08:37:54.771148 7119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:54 dist-test-slave-w6c0 krb5kdc[6509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584674, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:54.773415 7119 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260928 08:37:54.777782 7119 webserver.cc:460] Webserver started at http://127.0.97.60:35023/ using document root <none> and password file <none>
I20260928 08:37:54.777945 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44565
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42403
--webserver_port=38347
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:42403,127.0.97.61:38895
--tserver_master_addrs=127.0.97.62:42403,127.0.97.61:38895,127.0.97.60:40117 with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:37:54.778102 7119 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:54.778199 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:54.781366 7119 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 08:37:54.783546 7325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:54.784015 7119 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 08:37:54.784143 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/wal
uuid: "0ee6753344fc4449bad50ab34692b2ee"
format_stamp: "Formatted at 2026-09-28 08:37:54 on dist-test-slave-w6c0"
server_key: "f076bf4c1da216c89ee717b9fce100ec"
server_key_iv: "6fb26831860e5008fee2e4f7c82a7c03"
server_key_version: "encryptionkey@0"
I20260928 08:37:54.784358 7119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:54.802309 7119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:54.806820 7119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:54.842829 7119 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40117
I20260928 08:37:54.842881 7376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40117 every 8 connection(s)
I20260928 08:37:54.843999 7119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/master-2/data/info.pb
I20260928 08:37:54.849289 7377 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:37:54.853142 7377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: Bootstrap starting.
I20260928 08:37:54.877465 7377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:37:54.878019 7377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: Bootstrap complete.
I20260928 08:37:54.879091 7377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: true } }
I20260928 08:37:54.879382 7377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ee6753344fc4449bad50ab34692b2ee, State: Initialized, Role: LEARNER
I20260928 08:37:54.879643 7377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: true } }
I20260928 08:37:54.880430 7380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: true } } }
I20260928 08:37:54.880859 7380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [sys.catalog]: This master's current role is: LEARNER
I20260928 08:37:54.880869 7377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:37:54.889565 7391 catalog_manager.cc:1259] Loaded cluster ID: a053a569e8d440369b2fafe0808c6b64
I20260928 08:37:54.889657 7391 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: loading cluster ID for follower catalog manager: success
I20260928 08:37:54.893030 7391 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: acquiring CA information for follower catalog manager: success
I20260928 08:37:54.895371 7391 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 08:37:55.143254 7324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:55.143580 7324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:55.143750 7324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:55.151166 7324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:55.151304 7324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:55.151648 7324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:37:55.163050 7324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38347
--tserver_master_addrs=127.0.97.62:42403,127.0.97.61:38895,127.0.97.60:40117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:55.164386 7324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:55.165689 7324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:55.187273 7397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:55.187722 7396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:55.188767 7399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:55.189713 7324 server_base.cc:1034] running on GCE node
I20260928 08:37:55.190456 7324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:55.191521 7324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:55.192770 7324 hybrid_clock.cc:648] HybridClock initialized: now 1790584675192702 us; error 58 us; skew 500 ppm
I20260928 08:37:55.193122 7324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:55 dist-test-slave-w6c0 krb5kdc[6509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:55.199095 7324 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260928 08:37:55.202625 7324 webserver.cc:460] Webserver started at http://127.0.97.1:38347/ using document root <none> and password file <none>
I20260928 08:37:55.203435 7324 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:55.203601 7324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:55.209597 7324 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 08:37:55.233098 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:55.211160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40117 (local address 127.0.0.1:57052)
0928 08:37:55.219382 (+ 8222us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:55.219428 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:55.219635 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:55.223547 (+ 3912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:55.223561 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:55.223605 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:55.223899 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:55.223916 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:55.226619 (+ 2703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:55.226625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:55.229233 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:55.229256 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:55.231893 (+ 2637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:55.231904 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:55.232230 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:55.232294 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 08:37:55.232396 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 08:37:55.232608 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7938,"thread_start_us":15337,"threads_started":1}
I20260928 08:37:55.234309 7407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:55.211580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57052 (local address 127.0.97.60:40117)
0928 08:37:55.222986 (+ 11406us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:55.222996 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:55.223025 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:37:55.223114 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:55.223123 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:55.223243 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:55.224091 (+ 848us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:55.224354 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:55.226358 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:55.229518 (+ 3160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:55.231612 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:55.233506 (+ 1894us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:55.233748 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:55.233809 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 08:37:55.233953 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11222,"thread_start_us":11096,"threads_started":1}
I20260928 08:37:55.235363 7352 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 08:37:55.236231 7380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 899270827952449d802397379a7ed6ef. Latest consensus state: current_term: 1 leader_uuid: "899270827952449d802397379a7ed6ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: true } } }
I20260928 08:37:55.236552 7380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [sys.catalog]: This master's current role is: LEARNER
I20260928 08:37:55.237705 7408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:55.240038 7324 fs_manager.cc:725] Time spent opening block manager: real 0.028s user 0.008s sys 0.010s
I20260928 08:37:55.240429 7324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "f1478e5cb49148e4994ac79873f7a0ed"
format_stamp: "Formatted at 2026-09-28 08:37:52 on dist-test-slave-w6c0"
server_key: "6159ebb6c0902a366489f5984bc22424"
server_key_iv: "589e55637db35cf07047122ea1bf428a"
server_key_version: "encryptionkey@0"
I20260928 08:37:55.242422 7324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:55.278318 7324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:55.286393 7324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:55.286760 7324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:55.288282 7324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:55.290534 7324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:55.290668 7324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 08:37:55.290809 7324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:55.290908 7324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:55.345618 7324 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44565
I20260928 08:37:55.345650 7520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44565 every 8 connection(s)
I20260928 08:37:55.347702 7324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:37:55.352671 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 7324
I20260928 08:37:55.353024 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 6749
Sep 28 08:37:55 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Sep 28 08:37:55 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 28 08:37:55 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 28 08:37:55 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:37:55.386761 7407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:55.350952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34187 (local address 127.0.97.60:40117)
0928 08:37:55.351216 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:55.351226 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:55.358254 (+ 7028us) server_negotiation.cc:407] Connection header received
0928 08:37:55.359022 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:55.359031 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:55.359132 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:55.359212 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:55.368852 (+ 9640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:55.369922 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:55.372570 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:55.373247 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:55.373350 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:55.378687 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:55.378692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:55.378697 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:55.378738 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:55.382303 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:55.383768 (+ 1465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:55.383774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:55.383778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:55.383930 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:55.384544 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:55.384549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:55.384553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:55.384930 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:55.385113 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:55.386153 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0928 08:37:55.386401 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:37:55.388025 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:55.354303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47376 (local address 127.0.97.2:35875)
0928 08:37:55.358720 (+ 4417us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:55.358728 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:55.363205 (+ 4477us) server_negotiation.cc:407] Connection header received
0928 08:37:55.363304 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:55.363332 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:55.363629 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:55.363816 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:55.366435 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:55.368630 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:55.370543 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:55.371319 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:55.371449 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:55.378269 (+ 6820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:55.378283 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:55.378299 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:55.378341 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:55.382880 (+ 4539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:55.383976 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:55.383987 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:55.383990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:55.384143 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:55.385788 (+ 1645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:55.385794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:55.385798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:55.386188 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:55.386449 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:55.387426 (+ 977us) server_negotiation.cc:299] Negotiation successful
0928 08:37:55.387657 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4246,"thread_start_us":4300,"threads_started":1}
I20260928 08:37:55.389212 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:55.354867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40117 (local address 127.0.97.1:34187)
0928 08:37:55.358158 (+ 3291us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:55.358178 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:55.358833 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:55.359721 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:55.359736 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:55.360723 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:55.362216 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:55.362243 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:55.370106 (+ 7863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:55.370112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:55.372385 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:55.372400 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:55.373695 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:55.373703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:55.374005 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:55.374974 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:55.375015 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:55.378426 (+ 3411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:55.382491 (+ 4065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:55.382535 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:55.382557 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:55.383626 (+ 1069us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:55.384078 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:55.384083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:55.384086 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:55.384412 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:55.385075 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:55.385083 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:55.385641 (+ 558us) client_negotiation.cc:769] Sending connection context
0928 08:37:55.387317 (+ 1676us) client_negotiation.cc:240] Negotiation successful
0928 08:37:55.387794 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3165,"thread_start_us":3541,"threads_started":1}
I20260928 08:37:55.390089 6872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:55.390884 7521 heartbeater.cc:347] Connected to a master server at 127.0.97.60:40117
I20260928 08:37:55.391211 7521 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:55.392113 7521 heartbeater.cc:511] Master 127.0.97.60:40117 requested a full tablet report, sending...
I20260928 08:37:55.394574 7530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:55.358297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38895 (local address 127.0.97.1:60489)
0928 08:37:55.368408 (+ 10111us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:55.368427 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:55.368525 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:55.375362 (+ 6837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:55.375369 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:55.376477 (+ 1108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:55.376880 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:55.376894 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:55.378724 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:55.378731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:55.380208 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:55.380220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:55.381388 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:55.381393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:55.381568 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:55.382161 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:55.382183 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:55.386584 (+ 4401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:55.391679 (+ 5095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:55.391683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:55.391686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:55.392303 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:55.392742 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:55.392748 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:55.392753 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:55.392906 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:55.393595 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:55.393600 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:55.393740 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 08:37:55.393875 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 08:37:55.394139 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10002,"thread_start_us":10287,"threads_started":1}
I20260928 08:37:55.394488 7342 ts_manager.cc:194] Registered new tserver with Master: f1478e5cb49148e4994ac79873f7a0ed (127.0.97.1:44565)
I20260928 08:37:55.395665 7531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:55.358437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60489 (local address 127.0.97.61:38895)
0928 08:37:55.374899 (+ 16462us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:55.374906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:55.374930 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:37:55.374985 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:55.374990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:55.375085 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:55.377051 (+ 1966us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:55.377242 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:55.378571 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:55.380387 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:55.381069 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:55.381180 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:55.386790 (+ 5610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:55.386797 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:55.386802 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:55.386838 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:55.391524 (+ 4686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:55.392458 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:55.392465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:55.392468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:55.392608 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:55.393035 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:55.393040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:55.393043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:55.393390 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:55.393524 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:55.395075 (+ 1551us) server_negotiation.cc:299] Negotiation successful
0928 08:37:55.395303 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16329,"thread_start_us":16739,"threads_started":1}
I20260928 08:37:55.396533 7523 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38895
I20260928 08:37:55.396680 7523 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:55.397068 7523 heartbeater.cc:511] Master 127.0.97.61:38895 requested a full tablet report, sending...
I20260928 08:37:55.398541 7155 ts_manager.cc:194] Registered new tserver with Master: f1478e5cb49148e4994ac79873f7a0ed (127.0.97.1:44565)
I20260928 08:37:55.404034 7526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:55.351694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.1:59585)
0928 08:37:55.352711 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:55.352832 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:55.355722 (+ 2890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:55.374652 (+ 18930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:55.374657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:55.375107 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:55.375410 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:55.375427 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:55.379949 (+ 4522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:55.379955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:55.381383 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:55.381394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:55.382437 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:55.382444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:55.382691 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:55.383637 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:55.383658 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:55.386080 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:55.401162 (+ 15082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:55.401166 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:55.401168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:55.401479 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:55.401937 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:55.401944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:55.401951 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:55.402133 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:55.402866 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:55.402872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:55.403032 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 08:37:55.403173 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 08:37:55.403537 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":5475,"threads_started":1}
I20260928 08:37:55.405205 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:55.350645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59585 (local address 127.0.97.62:42403)
0928 08:37:55.368936 (+ 18291us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:55.368946 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:55.368972 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:37:55.369043 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:55.369051 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:55.369153 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:55.374628 (+ 5475us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:55.378609 (+ 3981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:55.379779 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:55.381571 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:55.382226 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:55.382973 (+ 747us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:55.386302 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:55.386310 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:55.386316 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:55.386357 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:55.401011 (+ 14654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:55.401617 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:55.401624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:55.401627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:55.401790 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:55.402288 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:55.402295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:55.402301 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:55.402725 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:55.404365 (+ 1640us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:55.404514 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 08:37:55.404801 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18117,"thread_start_us":18617,"threads_started":1}
I20260928 08:37:55.406013 7522 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42403
I20260928 08:37:55.406167 7522 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:55.406644 7522 heartbeater.cc:511] Master 127.0.97.62:42403 requested a full tablet report, sending...
I20260928 08:37:55.408094 6554 ts_manager.cc:194] Re-registered known tserver with Master: f1478e5cb49148e4994ac79873f7a0ed (127.0.97.1:44565)
I20260928 08:37:55.409446 6554 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59585
I20260928 08:37:55.486474 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 6749
I20260928 08:37:55.511665 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35875
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42403
--webserver_port=40947
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:42403,127.0.97.61:38895
--tserver_master_addrs=127.0.97.62:42403,127.0.97.61:38895,127.0.97.60:40117 with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:37:55.793181 7301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "899270827952449d802397379a7ed6ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: true } } }
I20260928 08:37:55.793499 7301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:55.795249 7535 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef: attempting to promote NON_VOTER 0ee6753344fc4449bad50ab34692b2ee to VOTER
I20260928 08:37:55.795780 7535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: false } }
I20260928 08:37:55.797771 7165 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 1 FOLLOWER]: Refusing update from remote peer 899270827952449d802397379a7ed6ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 08:37:55.797755 7352 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [term 1 LEARNER]: Refusing update from remote peer 899270827952449d802397379a7ed6ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 08:37:55.798338 7534 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 08:37:55.798765 7301 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 08:37:55.801343 7534 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0ee6753344fc4449bad50ab34692b2ee (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: false } } }
I20260928 08:37:55.801963 7165 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0ee6753344fc4449bad50ab34692b2ee (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: false } } }
I20260928 08:37:55.806443 7540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "899270827952449d802397379a7ed6ef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: false } } }
I20260928 08:37:55.806918 7540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:37:55.807173 7351 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0ee6753344fc4449bad50ab34692b2ee (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: false } } }
I20260928 08:37:55.808087 7301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "899270827952449d802397379a7ed6ef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: false } } }
I20260928 08:37:55.808151 7535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "899270827952449d802397379a7ed6ef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: false } } }
I20260928 08:37:55.808378 7535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:55.808377 7301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:55.808758 7301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "899270827952449d802397379a7ed6ef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: false } } }
I20260928 08:37:55.809012 7301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:55.811776 7541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "899270827952449d802397379a7ed6ef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: false } } }
I20260928 08:37:55.812083 7541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [sys.catalog]: This master's current role is: FOLLOWER
W20260928 08:37:55.863834 7533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:55.864135 7533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:55.864305 7533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:55.871563 7533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:55.871709 7533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:55.871884 7533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:37:55.882450 7533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40947
--tserver_master_addrs=127.0.97.62:42403,127.0.97.61:38895,127.0.97.60:40117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:55.883845 7533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:55.885320 7533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:55.899674 7546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:55.899827 7549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:55.899857 7547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:55.900880 7533 server_base.cc:1034] running on GCE node
I20260928 08:37:55.901564 7533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:55.902632 7533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:55.903872 7533 hybrid_clock.cc:648] HybridClock initialized: now 1790584675903835 us; error 32 us; skew 500 ppm
I20260928 08:37:55.904311 7533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:55 dist-test-slave-w6c0 krb5kdc[6509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:55.907680 7533 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260928 08:37:55.909197 7533 webserver.cc:460] Webserver started at http://127.0.97.2:40947/ using document root <none> and password file <none>
I20260928 08:37:55.909940 7533 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:55.910111 7533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:55.915498 7533 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:37:55.918044 7556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:55.919499 7533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:37:55.919679 7533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "5143bb66a241433383f2648dcb812a64"
format_stamp: "Formatted at 2026-09-28 08:37:52 on dist-test-slave-w6c0"
server_key: "7696d922de121ed599f7353a7b55fdb1"
server_key_iv: "7a27f3941206b035d33ccf8bca3578f3"
server_key_version: "encryptionkey@0"
I20260928 08:37:55.920725 7533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:55.942936 7533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:55.947189 7533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:55.947479 7533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:55.948395 7533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:55.949898 7533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:55.950018 7533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:55.950150 7533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:55.950237 7533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:55.984980 7533 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35875
I20260928 08:37:55.985066 7668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35875 every 8 connection(s)
I20260928 08:37:55.986469 7533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:37:55.996261 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 7533
I20260928 08:37:55.996577 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 6882
Sep 28 08:37:56 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 28 08:37:56 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 28 08:37:56 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Sep 28 08:37:56 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260928 08:37:56.034116 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:55.996372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53723 (local address 127.0.97.61:38895)
0928 08:37:56.014874 (+ 18502us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:56.014884 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:56.014908 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:37:56.014994 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:56.015001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:56.015101 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:56.015235 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:56.017118 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.018649 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.021685 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.022354 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.022530 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.026744 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:56.026750 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:56.026755 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:56.026801 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:56.030866 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.031795 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.031802 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.031808 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.032010 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.032442 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.032447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.032450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.032774 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:56.032915 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:56.033472 (+ 557us) server_negotiation.cc:299] Negotiation successful
0928 08:37:56.033705 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18337,"thread_start_us":20537,"threads_started":1}
I20260928 08:37:56.034672 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:55.989534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40441 (local address 127.0.97.62:42403)
0928 08:37:56.004997 (+ 15463us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:56.005007 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:56.007237 (+ 2230us) server_negotiation.cc:407] Connection header received
0928 08:37:56.007319 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:56.007339 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:56.007457 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:56.011868 (+ 4411us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:56.012112 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.013587 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.017479 (+ 3892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.018101 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.018221 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.024234 (+ 6013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:56.024239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:56.024242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:56.024279 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:56.029577 (+ 5298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.030618 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.030623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.030628 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.030771 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.031720 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.031728 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.031732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.032107 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:56.032324 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:56.034081 (+ 1757us) server_negotiation.cc:299] Negotiation successful
0928 08:37:56.034330 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15292,"thread_start_us":15479,"threads_started":1}
I20260928 08:37:56.035248 7680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:55.999326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38895 (local address 127.0.97.2:53723)
0928 08:37:56.013104 (+ 13778us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:56.013129 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:56.013275 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:56.015490 (+ 2215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:56.015502 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:56.016339 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:56.016790 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:56.016805 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.018850 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.018855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:56.021435 (+ 2580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:56.021457 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.022596 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.022603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:56.022780 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.023903 (+ 1123us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:56.023939 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:56.026536 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:56.031020 (+ 4484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:56.031026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:56.031032 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:56.031682 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:56.032144 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:56.032151 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:56.032156 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:56.032305 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:56.032939 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:56.032949 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:56.033098 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 08:37:56.033234 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 08:37:56.034595 (+ 1361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13617,"spinlock_wait_cycles":7808,"thread_start_us":14048,"threads_started":1}
I20260928 08:37:56.035624 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:55.990094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.2:40441)
0928 08:37:56.006913 (+ 16819us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:56.006939 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:56.007108 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:56.008569 (+ 1461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:56.008588 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:56.010390 (+ 1802us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:56.011623 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:56.011661 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.013824 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.013829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:56.016839 (+ 3010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:56.016863 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.018693 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.018703 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:56.019043 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.019899 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:56.019930 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:56.023995 (+ 4065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:56.029719 (+ 5724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:56.029725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:56.029738 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:56.030456 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:56.031230 (+ 774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:56.031239 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:56.031244 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:56.031520 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:56.032388 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:56.032396 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:56.032853 (+ 457us) client_negotiation.cc:769] Sending connection context
0928 08:37:56.033195 (+ 342us) client_negotiation.cc:240] Negotiation successful
0928 08:37:56.033735 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16629,"mutex_wait_us":407,"spinlock_wait_cycles":9984,"thread_start_us":5107,"threads_started":1}
I20260928 08:37:56.036993 7670 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42403
I20260928 08:37:56.037087 7671 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38895
I20260928 08:37:56.037318 7671 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:56.037324 7670 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:56.038408 7670 heartbeater.cc:511] Master 127.0.97.62:42403 requested a full tablet report, sending...
I20260928 08:37:56.039963 7671 heartbeater.cc:511] Master 127.0.97.61:38895 requested a full tablet report, sending...
I20260928 08:37:56.040503 6554 ts_manager.cc:194] Re-registered known tserver with Master: 5143bb66a241433383f2648dcb812a64 (127.0.97.2:35875)
I20260928 08:37:56.041038 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:55.990094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40117 (local address 127.0.97.2:33989)
0928 08:37:55.991240 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:55.991298 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:55.995145 (+ 3847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:56.023534 (+ 28389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:56.023542 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:56.024043 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:56.024371 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:56.024384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.025849 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.025857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:56.027535 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:56.027549 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.028656 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.028663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:56.028863 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.029860 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:56.029886 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:56.033063 (+ 3177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:56.038040 (+ 4977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:56.038047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:56.038053 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:56.038580 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:56.039030 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:56.039035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:56.039040 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:56.039289 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:56.039969 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:56.039973 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:56.040160 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 08:37:56.040313 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 08:37:56.040617 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":535,"threads_started":1}
I20260928 08:37:56.041543 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:55.989394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33989 (local address 127.0.97.60:40117)
0928 08:37:56.023155 (+ 33761us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:56.023163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:56.023186 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:37:56.023246 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:56.023274 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:56.023368 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:56.023503 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:56.024540 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.025694 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.027689 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.028428 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.029154 (+ 726us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.033261 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:56.033266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:56.033270 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:56.033305 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:56.037889 (+ 4584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.038748 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.038752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.038755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.038898 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.039422 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.039427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.039430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.039796 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:56.040816 (+ 1020us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:56.040946 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 08:37:56.041179 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33631,"thread_start_us":33524,"threads_started":1}
I20260928 08:37:56.041736 6554 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40441
I20260928 08:37:56.042258 7669 heartbeater.cc:347] Connected to a master server at 127.0.97.60:40117
I20260928 08:37:56.042382 7669 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:56.042928 7669 heartbeater.cc:511] Master 127.0.97.60:40117 requested a full tablet report, sending...
I20260928 08:37:56.043063 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:55.998138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56940 (local address 127.0.97.3:40031)
0928 08:37:56.008062 (+ 9924us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:56.008075 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:56.017718 (+ 9643us) server_negotiation.cc:407] Connection header received
0928 08:37:56.017799 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:56.017809 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:56.018087 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:56.018233 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:56.020702 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.022499 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.025280 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.026164 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.026285 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.034786 (+ 8501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:56.034803 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:56.034820 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:56.034862 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:56.039219 (+ 4357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.040140 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.040148 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.040152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.040304 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.040762 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.040769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.040775 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.041412 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:56.041921 (+ 509us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:56.042103 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 08:37:56.042582 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9729,"thread_start_us":9565,"threads_started":1}
I20260928 08:37:56.043267 7155 ts_manager.cc:194] Registered new tserver with Master: 5143bb66a241433383f2648dcb812a64 (127.0.97.2:35875)
I20260928 08:37:56.043992 7342 ts_manager.cc:194] Registered new tserver with Master: 5143bb66a241433383f2648dcb812a64 (127.0.97.2:35875)
I20260928 08:37:56.044610 7005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:56.127957 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 6882
I20260928 08:37:56.144748 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40031
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42403
--webserver_port=42645
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:42403,127.0.97.61:38895
--tserver_master_addrs=127.0.97.62:42403,127.0.97.61:38895,127.0.97.60:40117 with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:37:56.412693 7522 heartbeater.cc:503] Master 127.0.97.62:42403 was elected leader, sending a full tablet report...
W20260928 08:37:56.474802 7682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:37:56.475143 7682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:37:56.475238 7682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:37:56.482664 7682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:37:56.482777 7682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:37:56.482890 7682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:37:56.492844 7682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=42645
--tserver_master_addrs=127.0.97.62:42403,127.0.97.61:38895,127.0.97.60:40117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:37:56.494117 7682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:37:56.495394 7682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:37:56.508351 7688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:56.508461 7689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:37:56.508394 7691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:37:56.509131 7682 server_base.cc:1034] running on GCE node
I20260928 08:37:56.509696 7682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:37:56.510677 7682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:37:56.511909 7682 hybrid_clock.cc:648] HybridClock initialized: now 1790584676511872 us; error 50 us; skew 500 ppm
I20260928 08:37:56.512295 7682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:37:56 dist-test-slave-w6c0 krb5kdc[6509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584676, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:37:56.515316 7682 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260928 08:37:56.516693 7682 webserver.cc:460] Webserver started at http://127.0.97.3:42645/ using document root <none> and password file <none>
I20260928 08:37:56.517292 7682 fs_manager.cc:359] Metadata directory not provided
I20260928 08:37:56.517397 7682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:37:56.522325 7682 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:37:56.525022 7698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:56.526185 7682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:37:56.526353 7682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "3b24f682e35048979dadd681aaf430c6"
format_stamp: "Formatted at 2026-09-28 08:37:53 on dist-test-slave-w6c0"
server_key: "e0e313288c72e23afc4f05ecf26fb223"
server_key_iv: "cee885b63a5ef42636adaad2a80ba83d"
server_key_version: "encryptionkey@0"
I20260928 08:37:56.527339 7682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:37:56.552317 7682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:37:56.556136 7682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:37:56.556428 7682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:37:56.557338 7682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:37:56.558684 7682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:37:56.558799 7682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:56.558923 7682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:37:56.559016 7682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:37:56.591589 7682 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40031
I20260928 08:37:56.591676 7810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40031 every 8 connection(s)
I20260928 08:37:56.592971 7682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:37:56.595692 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 7682
Sep 28 08:37:56 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584676, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 28 08:37:56 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584676, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 28 08:37:56 dist-test-slave-w6c0 krb5kdc[6509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584676, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260928 08:37:56.637544 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:56.597218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58049 (local address 127.0.97.62:42403)
0928 08:37:56.603756 (+ 6538us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:56.603762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:56.603783 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:37:56.603833 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:56.603838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:56.603924 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:56.604104 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:56.605078 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.606184 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.609881 (+ 3697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.610600 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.610697 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.625774 (+ 15077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:56.625783 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:56.625791 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:56.625842 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:56.630968 (+ 5126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.632539 (+ 1571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.632546 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.632550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.632698 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.634328 (+ 1630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.634335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.634341 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.635143 (+ 802us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:56.635241 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:56.636921 (+ 1680us) server_negotiation.cc:299] Negotiation successful
0928 08:37:56.637138 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6427,"thread_start_us":6337,"threads_started":1}
I20260928 08:37:56.638090 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:56.596900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41145 (local address 127.0.97.60:40117)
0928 08:37:56.601129 (+ 4229us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:56.601135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:56.601154 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:37:56.601208 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:56.601213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:56.601292 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:56.601386 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:56.603788 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.605325 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.621424 (+ 16099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.622328 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.622474 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.629401 (+ 6927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:56.629407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:56.629413 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:56.629457 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:56.633564 (+ 4107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.634826 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.634835 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.634841 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.635502 (+ 661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.636112 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.636117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.636120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.636493 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:56.636655 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:56.637215 (+ 560us) server_negotiation.cc:299] Negotiation successful
0928 08:37:56.637562 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4111,"thread_start_us":4030,"threads_started":1}
I20260928 08:37:56.638857 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:56.596255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.3:58049)
0928 08:37:56.597072 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:56.597117 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:56.599841 (+ 2724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:56.604264 (+ 4423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:56.604268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:56.604704 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:56.604929 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:56.604939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.606336 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.606342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:56.609729 (+ 3387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:56.609743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.616874 (+ 7131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.616888 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:56.617352 (+ 464us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.619224 (+ 1872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:56.619269 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:56.625490 (+ 6221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:56.631939 (+ 6449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:56.631947 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:56.631970 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:56.632358 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:56.633620 (+ 1262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:56.633628 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:56.633634 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:56.633878 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:56.635964 (+ 2086us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:56.635970 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:56.636483 (+ 513us) client_negotiation.cc:769] Sending connection context
0928 08:37:56.636813 (+ 330us) client_negotiation.cc:240] Negotiation successful
0928 08:37:56.637344 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":1,"thread_start_us":368,"threads_started":1}
I20260928 08:37:56.639801 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:56.596256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40117 (local address 127.0.97.3:41145)
0928 08:37:56.600644 (+ 4388us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:56.600662 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:56.600763 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:56.601769 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:56.601779 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:56.602598 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:56.603330 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:56.603350 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.605580 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.605589 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:56.621208 (+ 15619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:56.621227 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.623889 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.623904 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:56.624141 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.625814 (+ 1673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:56.625845 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:56.629183 (+ 3338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:56.633816 (+ 4633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:56.633824 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:56.633830 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:56.634573 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:56.635682 (+ 1109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:56.635689 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:56.635695 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:56.635976 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:56.636628 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:56.636636 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:56.636837 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 08:37:56.638383 (+ 1546us) client_negotiation.cc:240] Negotiation successful
0928 08:37:56.638789 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4211,"mutex_wait_us":4071,"spinlock_wait_cycles":14080,"thread_start_us":4006,"threads_started":1}
I20260928 08:37:56.640851 7812 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42403
I20260928 08:37:56.640875 7811 heartbeater.cc:347] Connected to a master server at 127.0.97.60:40117
I20260928 08:37:56.641248 7812 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:56.641264 7811 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:56.642323 7812 heartbeater.cc:511] Master 127.0.97.62:42403 requested a full tablet report, sending...
I20260928 08:37:56.642359 7811 heartbeater.cc:511] Master 127.0.97.60:40117 requested a full tablet report, sending...
I20260928 08:37:56.644191 6554 ts_manager.cc:194] Re-registered known tserver with Master: 3b24f682e35048979dadd681aaf430c6 (127.0.97.3:40031)
I20260928 08:37:56.645409 6554 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58049
I20260928 08:37:56.646364 7342 ts_manager.cc:194] Registered new tserver with Master: 3b24f682e35048979dadd681aaf430c6 (127.0.97.3:40031)
I20260928 08:37:56.651239 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:56.597620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38895 (local address 127.0.97.3:54229)
0928 08:37:56.609360 (+ 11740us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:56.609380 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:56.609570 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:56.610116 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:56.610120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:56.618471 (+ 8351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:37:56.618880 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:56.618895 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.620755 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.620763 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:56.627598 (+ 6835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:56.627612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.629461 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.629467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:56.629669 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.632578 (+ 2909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:37:56.632599 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:37:56.641612 (+ 9013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:37:56.647889 (+ 6277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:56.647896 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:56.647901 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:56.648387 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:56.649203 (+ 816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:37:56.649208 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:37:56.649211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:37:56.649379 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:37:56.650133 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:37:56.650139 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:37:56.650282 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 08:37:56.650442 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 08:37:56.650814 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11596,"thread_start_us":10716,"threads_started":1}
I20260928 08:37:56.651296 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:56.597965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54229 (local address 127.0.97.61:38895)
0928 08:37:56.606903 (+ 8938us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:56.606909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:56.609468 (+ 2559us) server_negotiation.cc:407] Connection header received
0928 08:37:56.609720 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:56.609727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:56.609819 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:56.609900 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:56.619086 (+ 9186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.620535 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.627960 (+ 7425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.628602 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.628677 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.644115 (+ 15438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:56.644123 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:56.644128 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:56.644180 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:56.647705 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.648702 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.648707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.648711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.648873 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.649522 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.649529 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.649536 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.649912 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:56.650063 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:56.650610 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 08:37:56.650841 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8827,"thread_start_us":8741,"threads_started":1}
I20260928 08:37:56.652459 7813 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38895
I20260928 08:37:56.652598 7813 heartbeater.cc:464] Registering TS with master...
I20260928 08:37:56.653326 7813 heartbeater.cc:511] Master 127.0.97.61:38895 requested a full tablet report, sending...
I20260928 08:37:56.654470 7155 ts_manager.cc:194] Registered new tserver with Master: 3b24f682e35048979dadd681aaf430c6 (127.0.97.3:40031)
I20260928 08:37:56.661937 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:56.610758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57294 (local address 127.0.97.62:42403)
0928 08:37:56.611211 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:56.611220 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:56.634241 (+ 23021us) server_negotiation.cc:407] Connection header received
0928 08:37:56.634408 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:56.634412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:56.634482 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:56.634612 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:56.643979 (+ 9367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.646332 (+ 2353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.648809 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.649970 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.650079 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.655620 (+ 5541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:56.655629 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:56.655638 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:56.655666 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:56.658564 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.659362 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.659369 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.659374 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.659545 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.659965 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.659972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.659977 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.660325 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:56.660573 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:56.661295 (+ 722us) server_negotiation.cc:299] Negotiation successful
0928 08:37:56.661537 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":480,"threads_started":1}
I20260928 08:37:56.666101 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:56.611517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57062 (local address 127.0.97.60:40117)
0928 08:37:56.628431 (+ 16914us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:56.628438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:56.647738 (+ 19300us) server_negotiation.cc:407] Connection header received
0928 08:37:56.647852 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:56.647857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:56.647943 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:56.648013 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:56.651389 (+ 3376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.652481 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.654458 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.655128 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.655215 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.659177 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:56.659182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:56.659188 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:56.659223 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:56.662551 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.663166 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.663171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.663174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.663321 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.663688 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.663693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.663696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.664068 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:56.664302 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:56.665483 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0928 08:37:56.665693 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16790,"thread_start_us":16652,"threads_started":1}
I20260928 08:37:56.666746 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:56.631257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58356 (local address 127.0.97.61:38895)
0928 08:37:56.631833 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:56.631839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:56.649783 (+ 17944us) server_negotiation.cc:407] Connection header received
0928 08:37:56.649839 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:56.649843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:56.649922 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:56.650024 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:56.652302 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.653685 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.655718 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.656373 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.656493 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.660207 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:56.660212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:56.660215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:56.660241 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:56.663662 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.664463 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.664468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.664472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.664630 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:56.665019 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:56.665027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:56.665030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:56.665352 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:56.665584 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:56.666102 (+ 518us) server_negotiation.cc:299] Negotiation successful
0928 08:37:56.666323 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":303,"threads_started":1}
I20260928 08:37:56.666261 6554 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:37:56.673718 6554 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:37:56.738484 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:56.691574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35875 (local address 127.0.0.1:47378)
0928 08:37:56.729217 (+ 37643us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:56.729240 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:56.729385 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:56.732240 (+ 2855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:56.732244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:56.732288 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:56.732668 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:56.732678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.733959 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.733964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:56.735739 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:56.735774 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.737576 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.737619 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:56.737794 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.737833 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 08:37:56.737960 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 08:37:56.738085 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37484,"thread_start_us":37922,"threads_started":1}
I20260928 08:37:56.738461 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:56.691894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47378 (local address 127.0.97.2:35875)
0928 08:37:56.731732 (+ 39838us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:56.731740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:56.731774 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:37:56.731844 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:56.731850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:56.732077 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:56.732263 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:56.732851 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.733800 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.735946 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.737372 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.737509 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.737738 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:56.738001 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 08:37:56.738159 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39676,"thread_start_us":39540,"threads_started":1}
I20260928 08:37:56.741954 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:56.693149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40031 (local address 127.0.0.1:56954)
0928 08:37:56.730126 (+ 36977us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:56.730147 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:56.732488 (+ 2341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:56.733247 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:56.733251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:56.733277 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:56.733932 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:56.733940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.736470 (+ 2530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.736474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:56.738995 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:56.739010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.741041 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.741045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:56.741239 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.741298 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 08:37:56.741386 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 08:37:56.741524 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36821,"mutex_wait_us":38,"thread_start_us":34221,"threads_started":1}
I20260928 08:37:56.742056 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:56.691305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44565 (local address 127.0.0.1:37384)
0928 08:37:56.726333 (+ 35028us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:56.726368 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:56.726601 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:56.734210 (+ 7609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:56.734215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:56.734239 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:56.734677 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:56.734689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.736439 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.736445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:56.739029 (+ 2584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:56.739038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.740801 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.740810 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:56.740992 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.741026 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 08:37:56.741305 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 08:37:56.741631 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34797,"mutex_wait_us":75,"thread_start_us":38528,"threads_started":1}
I20260928 08:37:56.742594 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:56.693774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56954 (local address 127.0.97.3:40031)
0928 08:37:56.727319 (+ 33545us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:56.727334 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:56.730260 (+ 2926us) server_negotiation.cc:407] Connection header received
0928 08:37:56.732710 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:56.732718 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:56.732907 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:56.733019 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:56.734127 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.735788 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.739175 (+ 3387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.740604 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.740721 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.740906 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:56.741487 (+ 581us) server_negotiation.cc:299] Negotiation successful
0928 08:37:56.742311 (+ 824us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33287,"thread_start_us":34700,"threads_started":1}
I20260928 08:37:56.742813 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:56.691578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37384 (local address 127.0.97.1:44565)
0928 08:37:56.730744 (+ 39166us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:56.730753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:56.730778 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:37:56.730906 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:56.730912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:56.731178 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:56.734222 (+ 3044us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:56.734860 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.736313 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.739177 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:56.740595 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:56.740721 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:56.740911 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:56.742356 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0928 08:37:56.742537 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38997,"thread_start_us":38990,"threads_started":1}
I20260928 08:37:56.742856 7604 tablet_service.cc:1467] Processing CreateTablet for tablet 66dd9f8f0bf94296a33ae9e3fae9ad6a (DEFAULT_TABLE table=test-table [id=2bb465a400304942bf7870c1505e2b40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:56.744027 7604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66dd9f8f0bf94296a33ae9e3fae9ad6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:56.745872 7746 tablet_service.cc:1467] Processing CreateTablet for tablet 66dd9f8f0bf94296a33ae9e3fae9ad6a (DEFAULT_TABLE table=test-table [id=2bb465a400304942bf7870c1505e2b40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:56.747156 7746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66dd9f8f0bf94296a33ae9e3fae9ad6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:56.748071 7456 tablet_service.cc:1467] Processing CreateTablet for tablet 66dd9f8f0bf94296a33ae9e3fae9ad6a (DEFAULT_TABLE table=test-table [id=2bb465a400304942bf7870c1505e2b40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:56.749295 7456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66dd9f8f0bf94296a33ae9e3fae9ad6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:56.758742 7844 tablet_bootstrap.cc:492] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6: Bootstrap starting.
I20260928 08:37:56.759965 7845 tablet_bootstrap.cc:492] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64: Bootstrap starting.
I20260928 08:37:56.763538 7844 tablet_bootstrap.cc:654] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:56.763710 7845 tablet_bootstrap.cc:654] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:56.764961 7845 log.cc:824] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:56.765233 7844 log.cc:824] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:56.766973 7846 tablet_bootstrap.cc:492] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed: Bootstrap starting.
I20260928 08:37:56.768219 7845 tablet_bootstrap.cc:492] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64: No bootstrap required, opened a new log
I20260928 08:37:56.768523 7845 ts_tablet_manager.cc:1397] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 08:37:56.768705 7844 tablet_bootstrap.cc:492] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6: No bootstrap required, opened a new log
I20260928 08:37:56.768944 7844 ts_tablet_manager.cc:1397] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260928 08:37:56.770282 7846 tablet_bootstrap.cc:654] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:56.771462 7846 log.cc:824] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed: Log is configured to *not* fsync() on all Append() calls
I20260928 08:37:56.777410 7844 raft_consensus.cc:348] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } }
I20260928 08:37:56.777772 7844 raft_consensus.cc:374] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:56.777869 7844 raft_consensus.cc:729] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b24f682e35048979dadd681aaf430c6, State: Initialized, Role: FOLLOWER
I20260928 08:37:56.777961 7846 tablet_bootstrap.cc:492] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed: No bootstrap required, opened a new log
I20260928 08:37:56.778242 7846 ts_tablet_manager.cc:1397] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260928 08:37:56.778656 7844 consensus_queue.cc:260] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } }
I20260928 08:37:56.780078 7845 raft_consensus.cc:348] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } }
I20260928 08:37:56.780532 7845 raft_consensus.cc:374] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:56.780699 7845 raft_consensus.cc:729] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5143bb66a241433383f2648dcb812a64, State: Initialized, Role: FOLLOWER
I20260928 08:37:56.781399 7845 consensus_queue.cc:260] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } }
I20260928 08:37:56.791343 7846 raft_consensus.cc:348] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } }
I20260928 08:37:56.791750 7846 raft_consensus.cc:374] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:56.791857 7846 raft_consensus.cc:729] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1478e5cb49148e4994ac79873f7a0ed, State: Initialized, Role: FOLLOWER
I20260928 08:37:56.792073 7844 ts_tablet_manager.cc:1428] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6: Time spent starting tablet: real 0.023s user 0.010s sys 0.013s
I20260928 08:37:56.792510 7812 heartbeater.cc:503] Master 127.0.97.62:42403 was elected leader, sending a full tablet report...
I20260928 08:37:56.793396 7670 heartbeater.cc:503] Master 127.0.97.62:42403 was elected leader, sending a full tablet report...
I20260928 08:37:56.794096 7845 ts_tablet_manager.cc:1428] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64: Time spent starting tablet: real 0.025s user 0.010s sys 0.014s
I20260928 08:37:56.797288 7846 consensus_queue.cc:260] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } }
I20260928 08:37:56.800097 7846 ts_tablet_manager.cc:1428] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed: Time spent starting tablet: real 0.022s user 0.018s sys 0.000s
W20260928 08:37:56.845855 7814 tablet.cc:2369] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:37:56.858719 7524 tablet.cc:2369] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:37:56.989848 7672 tablet.cc:2369] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:37:57.042266 7852 raft_consensus.cc:482] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:37:57.042629 7852 raft_consensus.cc:504] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } }
I20260928 08:37:57.044227 7852 leader_election.cc:288] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5143bb66a241433383f2648dcb812a64 (127.0.97.2:35875), 3b24f682e35048979dadd681aaf430c6 (127.0.97.3:40031)
I20260928 08:37:57.053042 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:57.044651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55803 (local address 127.0.97.2:35875)
0928 08:37:57.044851 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:57.044858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:57.045963 (+ 1105us) server_negotiation.cc:407] Connection header received
0928 08:37:57.046174 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:57.046179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:57.046260 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:57.046351 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:57.047033 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:57.048053 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:57.050843 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:57.052134 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:57.052249 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:57.052332 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:57.052667 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 08:37:57.052802 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:37:57.053042 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:57.044646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59633 (local address 127.0.97.3:40031)
0928 08:37:57.044851 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:57.044858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:57.045717 (+ 859us) server_negotiation.cc:407] Connection header received
0928 08:37:57.045994 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:57.045998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:57.046057 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:57.046175 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:57.046725 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:57.047794 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:57.050804 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:57.052047 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:57.052190 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:57.052317 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:57.052667 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 08:37:57.052794 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:37:57.053298 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:57.044890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40031 (local address 127.0.97.1:59633)
0928 08:37:57.045579 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:57.045604 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:57.045825 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:57.046196 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:57.046199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:57.046218 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:57.046570 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:57.046586 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:57.047923 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:57.047927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:57.050649 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:57.050664 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:57.052262 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:57.052273 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:57.052498 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:57.052534 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 08:37:57.052629 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 08:37:57.052862 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":42,"thread_start_us":316,"threads_started":1}
I20260928 08:37:57.053340 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:57.044966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35875 (local address 127.0.97.1:55803)
0928 08:37:57.045875 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:57.045898 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:57.046043 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:57.046386 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:57.046390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:57.046408 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:57.046832 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:57.046845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:57.048174 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:57.048183 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:57.050688 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:57.050698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:57.052322 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:57.052326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:57.052512 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:57.052535 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 08:37:57.052629 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 08:37:57.052886 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":55,"thread_start_us":663,"threads_started":1}
I20260928 08:37:57.054410 7624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66dd9f8f0bf94296a33ae9e3fae9ad6a" candidate_uuid: "f1478e5cb49148e4994ac79873f7a0ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5143bb66a241433383f2648dcb812a64" is_pre_election: true
I20260928 08:37:57.054641 7766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66dd9f8f0bf94296a33ae9e3fae9ad6a" candidate_uuid: "f1478e5cb49148e4994ac79873f7a0ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b24f682e35048979dadd681aaf430c6" is_pre_election: true
I20260928 08:37:57.054977 7624 raft_consensus.cc:2454] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1478e5cb49148e4994ac79873f7a0ed in term 0.
I20260928 08:37:57.055092 7766 raft_consensus.cc:2454] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1478e5cb49148e4994ac79873f7a0ed in term 0.
I20260928 08:37:57.055708 7412 leader_election.cc:302] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b24f682e35048979dadd681aaf430c6, 5143bb66a241433383f2648dcb812a64, f1478e5cb49148e4994ac79873f7a0ed; no voters:
I20260928 08:37:57.056077 7852 raft_consensus.cc:2781] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:37:57.056239 7852 raft_consensus.cc:482] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:37:57.056349 7852 raft_consensus.cc:3037] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:57.058442 7852 raft_consensus.cc:504] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } }
I20260928 08:37:57.059072 7852 leader_election.cc:288] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [CANDIDATE]: Term 1 election: Requested vote from peers 5143bb66a241433383f2648dcb812a64 (127.0.97.2:35875), 3b24f682e35048979dadd681aaf430c6 (127.0.97.3:40031)
I20260928 08:37:57.059497 7766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66dd9f8f0bf94296a33ae9e3fae9ad6a" candidate_uuid: "f1478e5cb49148e4994ac79873f7a0ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b24f682e35048979dadd681aaf430c6"
I20260928 08:37:57.059508 7624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66dd9f8f0bf94296a33ae9e3fae9ad6a" candidate_uuid: "f1478e5cb49148e4994ac79873f7a0ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5143bb66a241433383f2648dcb812a64"
I20260928 08:37:57.059691 7766 raft_consensus.cc:3037] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:57.059697 7624 raft_consensus.cc:3037] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:57.061762 7766 raft_consensus.cc:2454] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1478e5cb49148e4994ac79873f7a0ed in term 1.
I20260928 08:37:57.062147 7410 leader_election.cc:302] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b24f682e35048979dadd681aaf430c6, f1478e5cb49148e4994ac79873f7a0ed; no voters:
I20260928 08:37:57.062407 7624 raft_consensus.cc:2454] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1478e5cb49148e4994ac79873f7a0ed in term 1.
I20260928 08:37:57.062431 7852 raft_consensus.cc:2781] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:57.062978 7852 raft_consensus.cc:686] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [term 1 LEADER]: Becoming Leader. State: Replica: f1478e5cb49148e4994ac79873f7a0ed, State: Running, Role: LEADER
I20260928 08:37:57.063726 7852 consensus_queue.cc:237] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } }
I20260928 08:37:57.068537 6554 catalog_manager.cc:5495] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed reported cstate change: term changed from 0 to 1, leader changed from <none> to f1478e5cb49148e4994ac79873f7a0ed (127.0.97.1). New cstate: current_term: 1 leader_uuid: "f1478e5cb49148e4994ac79873f7a0ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } health_report { overall_health: UNKNOWN } } }
I20260928 08:37:57.099427 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:57.094122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.0.97.1:44565)
0928 08:37:57.094325 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:57.094331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:57.094362 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:37:57.094471 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:57.094474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:57.094568 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:57.094718 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:37:57.095091 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:57.095966 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:57.097338 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:57.097737 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:57.097877 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:57.098310 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:37:57.098873 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:37:57.099047 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:57.099097 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 08:37:57.099175 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:37:57.107929 7766 raft_consensus.cc:1261] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6 [term 1 FOLLOWER]: Refusing update from remote peer f1478e5cb49148e4994ac79873f7a0ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:57.108002 7624 raft_consensus.cc:1261] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64 [term 1 FOLLOWER]: Refusing update from remote peer f1478e5cb49148e4994ac79873f7a0ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:57.108916 7852 consensus_queue.cc:1035] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:57.109215 7855 consensus_queue.cc:1035] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:57.123178 7858 mvcc.cc:205] Tried to move back new op lower bound from 7334234837424545792 to 7334234837256925184. Current Snapshot: MvccSnapshot[applied={T|T < 7334234837424545792}]
I20260928 08:37:57.124596 7860 mvcc.cc:205] Tried to move back new op lower bound from 7334234837424545792 to 7334234837256925184. Current Snapshot: MvccSnapshot[applied={T|T < 7334234837424545792}]
I20260928 08:37:57.139492 6553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:57.139737 6553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:57.145574 6553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 899270827952449d802397379a7ed6ef: Sending DeleteTablet for 3 replicas of tablet 66dd9f8f0bf94296a33ae9e3fae9ad6a
I20260928 08:37:57.146733 7604 tablet_service.cc:1514] Processing DeleteTablet for tablet 66dd9f8f0bf94296a33ae9e3fae9ad6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:57 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47378
I20260928 08:37:57.147068 7746 tablet_service.cc:1514] Processing DeleteTablet for tablet 66dd9f8f0bf94296a33ae9e3fae9ad6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:57 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56954
I20260928 08:37:57.162102 7456 tablet_service.cc:1514] Processing DeleteTablet for tablet 66dd9f8f0bf94296a33ae9e3fae9ad6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:57 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37384
I20260928 08:37:57.163324 7871 tablet_replica.cc:330] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed: stopping tablet replica
I20260928 08:37:57.163805 7871 raft_consensus.cc:2229] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:37:57.164556 7871 raft_consensus.cc:2258] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:57.166890 7871 ts_tablet_manager.cc:1905] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:57.174327 7871 ts_tablet_manager.cc:1918] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:37:57.174538 7871 log.cc:1196] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/wal/wals/66dd9f8f0bf94296a33ae9e3fae9ad6a
I20260928 08:37:57.175150 7871 ts_tablet_manager.cc:1939] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P f1478e5cb49148e4994ac79873f7a0ed: Deleting consensus metadata
I20260928 08:37:57.176990 6538 catalog_manager.cc:4841] TS f1478e5cb49148e4994ac79873f7a0ed (127.0.97.1:44565): tablet 66dd9f8f0bf94296a33ae9e3fae9ad6a (table test-table [id=2bb465a400304942bf7870c1505e2b40]) successfully deleted
I20260928 08:37:57.178967 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 6522
I20260928 08:37:57.179013 7869 tablet_replica.cc:330] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6: stopping tablet replica
I20260928 08:37:57.179603 7869 raft_consensus.cc:2229] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:57.180137 7869 raft_consensus.cc:2258] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:57.180214 6584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:57.181492 7868 tablet_replica.cc:330] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64: stopping tablet replica
I20260928 08:37:57.181942 7868 raft_consensus.cc:2229] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:57.182462 7868 raft_consensus.cc:2258] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:57.184158 7869 ts_tablet_manager.cc:1905] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:57.184774 7868 ts_tablet_manager.cc:1905] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:57.191610 7868 ts_tablet_manager.cc:1918] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:37:57.191829 7868 log.cc:1196] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/66dd9f8f0bf94296a33ae9e3fae9ad6a
I20260928 08:37:57.192416 7868 ts_tablet_manager.cc:1939] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 5143bb66a241433383f2648dcb812a64: Deleting consensus metadata
I20260928 08:37:57.198043 7869 ts_tablet_manager.cc:1918] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:37:57.198235 7869 log.cc:1196] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/66dd9f8f0bf94296a33ae9e3fae9ad6a
I20260928 08:37:57.198773 7869 ts_tablet_manager.cc:1939] T 66dd9f8f0bf94296a33ae9e3fae9ad6a P 3b24f682e35048979dadd681aaf430c6: Deleting consensus metadata
I20260928 08:37:57.202359 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:57.149963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58360 (local address 127.0.97.61:38895)
0928 08:37:57.150188 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:57.150197 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:57.171197 (+ 21000us) server_negotiation.cc:407] Connection header received
0928 08:37:57.178781 (+ 7584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:57.178787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:57.178877 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:57.178958 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:57.182785 (+ 3827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:57.183848 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:57.186360 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:57.187281 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:57.187398 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:57.191103 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:57.191111 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:57.191116 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:57.191154 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:57.199146 (+ 7992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:57.199961 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:57.199967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:57.199970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:57.200129 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:57.200584 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:57.200590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:57.200593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:57.200976 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:57.201198 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:57.201749 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 08:37:57.202011 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:37:57.220928 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:57.171884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57072 (local address 127.0.97.60:40117)
0928 08:37:57.177912 (+ 6028us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:57.177917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:57.204796 (+ 26879us) server_negotiation.cc:407] Connection header received
0928 08:37:57.204950 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:57.204956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:57.205047 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:57.205176 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:57.206908 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:57.208127 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:57.210367 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:57.211071 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:57.211220 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:57.214453 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:57.214460 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:57.214465 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:57.214537 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:57.218110 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:57.218878 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:57.218883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:57.218888 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:57.219041 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:57.219457 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:57.219463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:57.219469 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:57.219799 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:57.220053 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:57.220342 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 08:37:57.220559 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5918,"thread_start_us":5808,"threads_started":1}
I20260928 08:37:57.315251 6539 catalog_manager.cc:4841] TS 3b24f682e35048979dadd681aaf430c6 (127.0.97.3:40031): tablet 66dd9f8f0bf94296a33ae9e3fae9ad6a (table test-table [id=2bb465a400304942bf7870c1505e2b40]) successfully deleted
I20260928 08:37:57.315578 6541 catalog_manager.cc:4841] TS 5143bb66a241433383f2648dcb812a64 (127.0.97.2:35875): tablet 66dd9f8f0bf94296a33ae9e3fae9ad6a (table test-table [id=2bb465a400304942bf7870c1505e2b40]) successfully deleted
I20260928 08:37:57.315824 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 6522
I20260928 08:37:57.363597 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:57.343638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58364 (local address 127.0.97.61:38895)
0928 08:37:57.343823 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:57.343831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:57.344497 (+ 666us) server_negotiation.cc:407] Connection header received
0928 08:37:57.344567 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:57.344571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:57.344671 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:57.344762 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:57.347110 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:57.348221 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:57.352126 (+ 3905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:57.352822 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:57.352919 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:57.356651 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:57.356658 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:57.356663 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:57.356700 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:57.360588 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:57.361373 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:57.361379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:57.361383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:57.361526 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:57.361939 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:57.361944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:57.361948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:57.362258 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:57.362461 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:57.363022 (+ 561us) server_negotiation.cc:299] Negotiation successful
0928 08:37:57.363258 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:37:57.364897 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:57.342573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57084 (local address 127.0.97.60:40117)
0928 08:37:57.348050 (+ 5477us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:57.348056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:57.348074 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:37:57.348132 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:57.348137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:57.348204 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:57.348298 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:57.351027 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:57.352267 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:57.354324 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:57.354996 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:57.355692 (+ 696us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:57.358116 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:57.358122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:57.358127 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:57.358164 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:57.361439 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:57.362252 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:57.362257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:57.362261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:57.362392 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:57.362831 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:57.362844 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:57.362847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:57.363148 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:57.364230 (+ 1082us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:57.364347 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 08:37:57.364552 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5322}
I20260928 08:37:58.127053 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.126235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.1:43171)
0928 08:37:58.126787 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.126893 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":200,"threads_started":1}
W20260928 08:37:58.127488 7522 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42403 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
I20260928 08:37:58.130897 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.129935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.2:52917)
0928 08:37:58.130481 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.130638 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":313,"threads_started":1}
I20260928 08:37:58.131307 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.130440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.3:36989)
0928 08:37:58.131002 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.131130 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":308,"threads_started":1}
W20260928 08:37:58.131788 7670 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42403 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
W20260928 08:37:58.131955 7812 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42403 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
I20260928 08:37:58.646330 7884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 899270827952449d802397379a7ed6ef)
I20260928 08:37:58.646628 7884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: false } }
I20260928 08:37:58.647902 7884 leader_election.cc:288] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 899270827952449d802397379a7ed6ef (127.0.97.62:42403), 2590393e2ec44ddfa136fbc2c1f17b79 (127.0.97.61:38895)
I20260928 08:37:58.649139 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.648040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.0.1:57720)
0928 08:37:58.648541 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.648727 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":245,"threads_started":1}
W20260928 08:37:58.649443 7326 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
I20260928 08:37:58.651124 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.650735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.0.1:57732)
0928 08:37:58.650898 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.650987 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 08:37:58.651274 7326 leader_election.cc:334] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 899270827952449d802397379a7ed6ef (127.0.97.62:42403): Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
I20260928 08:37:58.654919 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.648619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38895 (local address 127.0.0.1:58376)
0928 08:37:58.649237 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.649255 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:58.649375 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:58.649886 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:58.649893 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:58.649928 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:58.650359 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:58.650369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.651756 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.651762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:58.653052 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:58.653063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.654152 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.654158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:58.654321 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:58.654346 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 08:37:58.654451 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 08:37:58.654594 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":242,"threads_started":1}
I20260928 08:37:58.654954 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.648765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58376 (local address 127.0.97.61:38895)
0928 08:37:58.649392 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:58.649399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:58.649424 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:37:58.649493 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:58.649500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:58.649583 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:58.649686 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:58.650533 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.651612 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.653203 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.653980 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.654124 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:58.654220 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:58.654475 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 08:37:58.654636 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":321,"threads_started":1}
I20260928 08:37:58.655553 7165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ee6753344fc4449bad50ab34692b2ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" is_pre_election: true
I20260928 08:37:58.656273 7326 leader_election.cc:302] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ee6753344fc4449bad50ab34692b2ee; no voters: 2590393e2ec44ddfa136fbc2c1f17b79, 899270827952449d802397379a7ed6ef
I20260928 08:37:58.656605 7884 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:37:58.662000 7889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 899270827952449d802397379a7ed6ef)
I20260928 08:37:58.662194 7889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: false } }
I20260928 08:37:58.663594 7889 leader_election.cc:288] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 899270827952449d802397379a7ed6ef (127.0.97.62:42403), 0ee6753344fc4449bad50ab34692b2ee (127.0.97.60:40117)
I20260928 08:37:58.664746 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.663659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.0.1:57738)
0928 08:37:58.664232 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.664440 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":286,"threads_started":1}
W20260928 08:37:58.665046 7139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
I20260928 08:37:58.666615 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.666238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.0.1:57744)
0928 08:37:58.666385 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.666450 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 08:37:58.666813 7139 leader_election.cc:334] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 899270827952449d802397379a7ed6ef (127.0.97.62:42403): Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
I20260928 08:37:58.669834 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.663807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40117 (local address 127.0.0.1:57092)
0928 08:37:58.664566 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.664583 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:58.664712 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:58.665200 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:58.665206 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:58.665231 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:58.665667 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:58.665680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.667014 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.667019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:58.668244 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:58.668255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.669296 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.669300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:58.669427 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:58.669450 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 08:37:58.669543 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 08:37:58.669629 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":512,"threads_started":1}
I20260928 08:37:58.669950 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.664003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57092 (local address 127.0.97.60:40117)
0928 08:37:58.664948 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:58.664952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:58.664970 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:37:58.665017 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:58.665020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:58.665069 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:58.665150 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:58.665864 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.666854 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.668386 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.669126 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.669232 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:58.669307 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:58.669574 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 08:37:58.669676 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":713,"threads_started":1}
I20260928 08:37:58.670514 7351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ee6753344fc4449bad50ab34692b2ee" is_pre_election: true
I20260928 08:37:58.670827 7351 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2590393e2ec44ddfa136fbc2c1f17b79 in term 1.
I20260928 08:37:58.671315 7141 leader_election.cc:302] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ee6753344fc4449bad50ab34692b2ee, 2590393e2ec44ddfa136fbc2c1f17b79; no voters: 899270827952449d802397379a7ed6ef
I20260928 08:37:58.671725 7889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 08:37:58.671847 7889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 899270827952449d802397379a7ed6ef)
I20260928 08:37:58.671952 7889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:37:58.673969 7889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: false } }
I20260928 08:37:58.674593 7889 leader_election.cc:288] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [CANDIDATE]: Term 2 election: Requested vote from peers 899270827952449d802397379a7ed6ef (127.0.97.62:42403), 0ee6753344fc4449bad50ab34692b2ee (127.0.97.60:40117)
I20260928 08:37:58.675087 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.674652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.0.1:57746)
0928 08:37:58.674873 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.674990 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260928 08:37:58.675048 7351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ee6753344fc4449bad50ab34692b2ee"
I20260928 08:37:58.675258 7351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:37:58.676098 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.675806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.0.1:57752)
0928 08:37:58.675939 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.675998 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 08:37:58.676340 7139 leader_election.cc:334] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 899270827952449d802397379a7ed6ef (127.0.97.62:42403): Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
I20260928 08:37:58.677609 7351 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2590393e2ec44ddfa136fbc2c1f17b79 in term 2.
I20260928 08:37:58.677955 7141 leader_election.cc:302] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ee6753344fc4449bad50ab34692b2ee, 2590393e2ec44ddfa136fbc2c1f17b79; no voters: 899270827952449d802397379a7ed6ef
I20260928 08:37:58.678231 7889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:37:58.678681 7889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [term 2 LEADER]: Becoming Leader. State: Replica: 2590393e2ec44ddfa136fbc2c1f17b79, State: Running, Role: LEADER
I20260928 08:37:58.679044 7889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: false } }
I20260928 08:37:58.680830 7894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2590393e2ec44ddfa136fbc2c1f17b79. Latest consensus state: current_term: 2 leader_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: false } } }
I20260928 08:37:58.681283 7894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:58.681923 7896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:37:58.685607 7896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:37:58.686394 7896 catalog_manager.cc:1259] Loaded cluster ID: a053a569e8d440369b2fafe0808c6b64
I20260928 08:37:58.686462 7896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:37:58.687561 7896 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:37:58.688287 7896 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: Loaded TSK: 0
I20260928 08:37:58.689059 7896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:37:58.701929 7154 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:37:58.708776 7154 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:37:58.712756 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.712328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.0.1:57768)
0928 08:37:58.712534 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.712616 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 08:37:58.712823 7351 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [term 2 FOLLOWER]: Refusing update from remote peer 2590393e2ec44ddfa136fbc2c1f17b79: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 08:37:58.713572 7894 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 08:37:58.713701 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.713328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.0.1:57778)
0928 08:37:58.713510 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.713582 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 08:37:58.713976 7139 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 -> Peer 899270827952449d802397379a7ed6ef (127.0.97.62:42403): Couldn't send request to peer 899270827952449d802397379a7ed6ef. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 08:37:58.716392 7884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2590393e2ec44ddfa136fbc2c1f17b79. Latest consensus state: current_term: 2 leader_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: false } } }
I20260928 08:37:58.716735 7884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:37:58.717819 7894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: false } } }
I20260928 08:37:58.718129 7894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79 [sys.catalog]: This master's current role is: LEADER
I20260928 08:37:58.718667 7884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "899270827952449d802397379a7ed6ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42403 } } peers { permanent_uuid: "2590393e2ec44ddfa136fbc2c1f17b79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38895 } attrs { promote: false } } peers { permanent_uuid: "0ee6753344fc4449bad50ab34692b2ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40117 } attrs { promote: false } } }
I20260928 08:37:58.718945 7884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ee6753344fc4449bad50ab34692b2ee [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:37:58.749960 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.726135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44565 (local address 127.0.0.1:37396)
0928 08:37:58.726337 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.726358 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:58.726474 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:58.741473 (+ 14999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:58.741482 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:58.741521 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:58.741833 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:58.741849 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.743985 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.743994 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:58.746700 (+ 2706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:58.746720 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.748882 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.748891 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:58.749169 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:58.749228 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 08:37:58.749312 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 08:37:58.749534 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 08:37:58.752164 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.726259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.0.97.1:44565)
0928 08:37:58.740829 (+ 14570us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:58.740839 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:58.740869 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:37:58.740969 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:58.740977 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:58.741143 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:58.742133 (+ 990us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:58.742346 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.743767 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.746962 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.748639 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.750827 (+ 2188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:58.750982 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:58.751059 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 08:37:58.751871 (+ 812us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14378,"thread_start_us":15026,"threads_started":1}
I20260928 08:37:58.752892 7456 tablet_service.cc:1467] Processing CreateTablet for tablet 65f678544c88419cbe820d11dda2b2f4 (DEFAULT_TABLE table=test-table [id=8e0caeaa3eb34ed4beefc55e3f7a4efc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:58.753384 7456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f678544c88419cbe820d11dda2b2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:58.757162 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.727397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35875 (local address 127.0.0.1:47388)
0928 08:37:58.750271 (+ 22874us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.750296 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:58.750422 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:58.751325 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:58.751333 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:58.751369 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:58.751635 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:58.751649 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.753807 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.753815 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:58.755251 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:58.755262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.756338 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.756343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:58.756535 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:58.756566 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 08:37:58.756675 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 08:37:58.756801 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22743,"mutex_wait_us":23,"thread_start_us":24187,"threads_started":1}
I20260928 08:37:58.757886 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.727974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47388 (local address 127.0.97.2:35875)
0928 08:37:58.750946 (+ 22972us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:58.750956 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:58.750982 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:37:58.751069 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:58.751077 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:58.751176 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:58.751310 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:58.752650 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.753674 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.755414 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.756150 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.756309 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:58.756409 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:58.757459 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0928 08:37:58.757600 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22807,"thread_start_us":22709,"threads_started":1}
I20260928 08:37:58.758659 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.727188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40031 (local address 127.0.0.1:56970)
0928 08:37:58.728292 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.728321 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:58.728478 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:58.752396 (+ 23918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:58.752403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:58.752429 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:58.752746 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:58.752756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.754268 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.754273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:58.756352 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:58.756367 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.757965 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.757971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:58.758149 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:58.758179 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 08:37:58.758245 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 08:37:58.758391 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":4}
I20260928 08:37:58.758699 7604 tablet_service.cc:1467] Processing CreateTablet for tablet 65f678544c88419cbe820d11dda2b2f4 (DEFAULT_TABLE table=test-table [id=8e0caeaa3eb34ed4beefc55e3f7a4efc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:58.759095 7604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f678544c88419cbe820d11dda2b2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:58.759297 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.727559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56970 (local address 127.0.97.3:40031)
0928 08:37:58.752121 (+ 24562us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:58.752127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:58.752143 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:37:58.752198 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:58.752201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:58.752253 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:58.752380 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:58.752927 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.754117 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.756548 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.757738 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.758641 (+ 903us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:58.758758 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:58.758815 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 08:37:58.758975 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24466,"thread_start_us":24380,"threads_started":1}
I20260928 08:37:58.759860 7908 tablet_bootstrap.cc:492] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed: Bootstrap starting.
I20260928 08:37:58.761179 7908 tablet_bootstrap.cc:654] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:58.763345 7908 tablet_bootstrap.cc:492] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed: No bootstrap required, opened a new log
I20260928 08:37:58.763486 7908 ts_tablet_manager.cc:1397] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 08:37:58.764264 7908 raft_consensus.cc:348] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } }
I20260928 08:37:58.764477 7908 raft_consensus.cc:374] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:58.764586 7908 raft_consensus.cc:729] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1478e5cb49148e4994ac79873f7a0ed, State: Initialized, Role: FOLLOWER
I20260928 08:37:58.764837 7908 consensus_queue.cc:260] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } }
I20260928 08:37:58.765007 7910 tablet_bootstrap.cc:492] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64: Bootstrap starting.
I20260928 08:37:58.765208 7746 tablet_service.cc:1467] Processing CreateTablet for tablet 65f678544c88419cbe820d11dda2b2f4 (DEFAULT_TABLE table=test-table [id=8e0caeaa3eb34ed4beefc55e3f7a4efc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:37:58.765605 7746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f678544c88419cbe820d11dda2b2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:37:58.766759 7910 tablet_bootstrap.cc:654] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:58.769068 7910 tablet_bootstrap.cc:492] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64: No bootstrap required, opened a new log
I20260928 08:37:58.769222 7910 ts_tablet_manager.cc:1397] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 08:37:58.770193 7910 raft_consensus.cc:348] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } }
I20260928 08:37:58.770491 7910 raft_consensus.cc:374] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:58.770634 7910 raft_consensus.cc:729] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5143bb66a241433383f2648dcb812a64, State: Initialized, Role: FOLLOWER
I20260928 08:37:58.770959 7910 consensus_queue.cc:260] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } }
I20260928 08:37:58.771823 7908 ts_tablet_manager.cc:1428] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed: Time spent starting tablet: real 0.008s user 0.003s sys 0.006s
I20260928 08:37:58.772712 7913 tablet_bootstrap.cc:492] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6: Bootstrap starting.
I20260928 08:37:58.774441 7913 tablet_bootstrap.cc:654] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6: Neither blocks nor log segments found. Creating new log.
I20260928 08:37:58.779136 7913 tablet_bootstrap.cc:492] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6: No bootstrap required, opened a new log
I20260928 08:37:58.779281 7913 ts_tablet_manager.cc:1397] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 08:37:58.779529 7910 ts_tablet_manager.cc:1428] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64: Time spent starting tablet: real 0.010s user 0.001s sys 0.004s
I20260928 08:37:58.780066 7913 raft_consensus.cc:348] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } }
I20260928 08:37:58.780310 7913 raft_consensus.cc:374] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:37:58.780416 7913 raft_consensus.cc:729] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b24f682e35048979dadd681aaf430c6, State: Initialized, Role: FOLLOWER
I20260928 08:37:58.780651 7913 consensus_queue.cc:260] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } }
I20260928 08:37:58.785748 7523 heartbeater.cc:503] Master 127.0.97.61:38895 was elected leader, sending a full tablet report...
I20260928 08:37:58.786144 7913 ts_tablet_manager.cc:1428] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20260928 08:37:58.787393 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.774456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.1:44937)
0928 08:37:58.785644 (+ 11188us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.785792 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10984,"thread_start_us":10754,"threads_started":1}
I20260928 08:37:58.787562 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.786759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.3:41017)
0928 08:37:58.787247 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.787350 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"threads_started":1}
I20260928 08:37:58.789600 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.789203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.1:45873)
0928 08:37:58.789398 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.789473 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 08:37:58.790875 7813 heartbeater.cc:503] Master 127.0.97.61:38895 was elected leader, sending a full tablet report...
I20260928 08:37:58.790885 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.789734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.3:51473)
0928 08:37:58.790664 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.790743 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":834}
I20260928 08:37:58.792086 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.776394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.2:34327)
0928 08:37:58.791786 (+ 15392us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.791890 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15277,"threads_started":1}
I20260928 08:37:58.793282 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.792989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.2:45495)
0928 08:37:58.793127 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.793187 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 08:37:58.793812 7671 heartbeater.cc:503] Master 127.0.97.61:38895 was elected leader, sending a full tablet report...
I20260928 08:37:58.809369 7918 raft_consensus.cc:482] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:37:58.809628 7918 raft_consensus.cc:504] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } }
I20260928 08:37:58.811451 7918 leader_election.cc:288] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1478e5cb49148e4994ac79873f7a0ed (127.0.97.1:44565), 5143bb66a241433383f2648dcb812a64 (127.0.97.2:35875)
I20260928 08:37:58.817868 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.811717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60341 (local address 127.0.97.1:44565)
0928 08:37:58.811914 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:58.811921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:58.811943 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:37:58.812162 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:58.812178 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:58.812269 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:58.812380 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:58.812902 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.813818 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.815774 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.816958 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.817086 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:58.817200 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:58.817502 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 08:37:58.817597 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 08:37:58.817932 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.811590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44565 (local address 127.0.97.3:60341)
0928 08:37:58.811845 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.811867 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:58.811994 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:58.812424 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:58.812432 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:58.812483 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:58.812759 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:58.812771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.813958 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.813965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:58.815591 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:58.815603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.817129 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.817133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:58.817311 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:58.817362 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 08:37:58.817467 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 08:37:58.817599 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":62}
I20260928 08:37:58.818243 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.811916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38107 (local address 127.0.97.2:35875)
0928 08:37:58.812158 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:58.812167 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:58.812652 (+ 485us) server_negotiation.cc:407] Connection header received
0928 08:37:58.812773 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:58.812778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:58.812860 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:58.812951 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:37:58.813439 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.814237 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.816027 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.817193 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.817316 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:58.817380 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:58.817710 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 08:37:58.817839 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:37:58.818339 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.811821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35875 (local address 127.0.97.3:38107)
0928 08:37:58.812549 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.812565 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:37:58.812659 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:37:58.812986 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:37:58.812992 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:37:58.813017 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:37:58.813288 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:58.813301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.814368 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.814372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:58.815866 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:37:58.815880 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.817368 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.817373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:37:58.817556 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:58.817585 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 08:37:58.817660 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 08:37:58.817790 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":430,"threads_started":1}
I20260928 08:37:58.818904 7624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f678544c88419cbe820d11dda2b2f4" candidate_uuid: "3b24f682e35048979dadd681aaf430c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5143bb66a241433383f2648dcb812a64" is_pre_election: true
I20260928 08:37:58.818837 7476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f678544c88419cbe820d11dda2b2f4" candidate_uuid: "3b24f682e35048979dadd681aaf430c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1478e5cb49148e4994ac79873f7a0ed" is_pre_election: true
I20260928 08:37:58.819133 7624 raft_consensus.cc:2454] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b24f682e35048979dadd681aaf430c6 in term 0.
I20260928 08:37:58.819198 7476 raft_consensus.cc:2454] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b24f682e35048979dadd681aaf430c6 in term 0.
I20260928 08:37:58.819703 7702 leader_election.cc:302] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b24f682e35048979dadd681aaf430c6, 5143bb66a241433383f2648dcb812a64, f1478e5cb49148e4994ac79873f7a0ed; no voters:
I20260928 08:37:58.820041 7918 raft_consensus.cc:2781] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:37:58.820158 7918 raft_consensus.cc:482] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:37:58.820276 7918 raft_consensus.cc:3037] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:58.822343 7918 raft_consensus.cc:504] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } }
I20260928 08:37:58.822947 7918 leader_election.cc:288] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [CANDIDATE]: Term 1 election: Requested vote from peers f1478e5cb49148e4994ac79873f7a0ed (127.0.97.1:44565), 5143bb66a241433383f2648dcb812a64 (127.0.97.2:35875)
I20260928 08:37:58.823344 7476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f678544c88419cbe820d11dda2b2f4" candidate_uuid: "3b24f682e35048979dadd681aaf430c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1478e5cb49148e4994ac79873f7a0ed"
I20260928 08:37:58.823534 7476 raft_consensus.cc:3037] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:58.823451 7624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f678544c88419cbe820d11dda2b2f4" candidate_uuid: "3b24f682e35048979dadd681aaf430c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5143bb66a241433383f2648dcb812a64"
I20260928 08:37:58.823681 7624 raft_consensus.cc:3037] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:37:58.825745 7624 raft_consensus.cc:2454] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b24f682e35048979dadd681aaf430c6 in term 1.
I20260928 08:37:58.825776 7476 raft_consensus.cc:2454] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b24f682e35048979dadd681aaf430c6 in term 1.
I20260928 08:37:58.826122 7702 leader_election.cc:302] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b24f682e35048979dadd681aaf430c6, 5143bb66a241433383f2648dcb812a64; no voters:
I20260928 08:37:58.826491 7918 raft_consensus.cc:2781] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:37:58.826968 7918 raft_consensus.cc:686] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [term 1 LEADER]: Becoming Leader. State: Replica: 3b24f682e35048979dadd681aaf430c6, State: Running, Role: LEADER
I20260928 08:37:58.827663 7918 consensus_queue.cc:237] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } }
I20260928 08:37:58.827885 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.827516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.3:54211)
0928 08:37:58.827653 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.827757 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 08:37:58.831024 7155 catalog_manager.cc:5495] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b24f682e35048979dadd681aaf430c6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "3b24f682e35048979dadd681aaf430c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b24f682e35048979dadd681aaf430c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40031 } health_report { overall_health: HEALTHY } } }
I20260928 08:37:58.857475 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.852728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56976 (local address 127.0.97.3:40031)
0928 08:37:58.852912 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:58.852917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:58.852934 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:37:58.853169 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:58.853174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:58.853242 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:58.853357 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:37:58.853711 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.854403 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.855616 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.856022 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.856110 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:58.856436 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:37:58.856945 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:37:58.857054 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:58.857126 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 08:37:58.857186 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:37:58.861219 7624 raft_consensus.cc:1261] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64 [term 1 FOLLOWER]: Refusing update from remote peer 3b24f682e35048979dadd681aaf430c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:58.861521 7476 raft_consensus.cc:1261] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed [term 1 FOLLOWER]: Refusing update from remote peer 3b24f682e35048979dadd681aaf430c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:37:58.861919 7921 consensus_queue.cc:1035] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5143bb66a241433383f2648dcb812a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:58.862264 7918 consensus_queue.cc:1035] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1478e5cb49148e4994ac79873f7a0ed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:37:58.864948 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.864354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.2:45767)
0928 08:37:58.864578 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.864683 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260928 08:37:58.865025 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.864548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.1:55207)
0928 08:37:58.864776 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.864866 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260928 08:37:58.871323 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.870887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.3:47645)
0928 08:37:58.871085 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.871172 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 08:37:58.874675 7924 mvcc.cc:205] Tried to move back new op lower bound from 7334234844609380352 to 7334234844482060288. Current Snapshot: MvccSnapshot[applied={T|T < 7334234844609380352 or (T in {7334234844609380352})}]
I20260928 08:37:58.875981 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.875411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.2:48391)
0928 08:37:58.875673 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.875779 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260928 08:37:58.884068 7155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:58.884303 7155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:37:58.886296 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.884484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.3:37899)
0928 08:37:58.886006 (+ 1522us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.886092 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1421}
I20260928 08:37:58.887288 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.886826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.97.1:45195)
0928 08:37:58.887058 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:58.887144 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260928 08:37:58.891067 7155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2590393e2ec44ddfa136fbc2c1f17b79: Sending DeleteTablet for 3 replicas of tablet 65f678544c88419cbe820d11dda2b2f4
I20260928 08:37:58.892122 7604 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f678544c88419cbe820d11dda2b2f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:58 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:47388
I20260928 08:37:58.892503 7456 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f678544c88419cbe820d11dda2b2f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:58 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:37396
I20260928 08:37:58.892804 7933 tablet_replica.cc:330] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64: stopping tablet replica
I20260928 08:37:58.892985 7933 raft_consensus.cc:2229] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:58.893271 7933 raft_consensus.cc:2258] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:58.893764 7746 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f678544c88419cbe820d11dda2b2f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:58 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:56970
I20260928 08:37:58.896116 7745 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f678544c88419cbe820d11dda2b2f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:37:58 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:56970
I20260928 08:37:58.896621 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 7324
I20260928 08:37:58.900544 7933 ts_tablet_manager.cc:1905] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:58.903744 7934 tablet_replica.cc:330] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed: stopping tablet replica
I20260928 08:37:58.904150 7934 raft_consensus.cc:2229] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:37:58.904588 7934 raft_consensus.cc:2258] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:58.906000 7934 ts_tablet_manager.cc:1905] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:58.912974 7934 ts_tablet_manager.cc:1918] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:37:58.913178 7934 log.cc:1196] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-0/wal/wals/65f678544c88419cbe820d11dda2b2f4
I20260928 08:37:58.913882 7934 ts_tablet_manager.cc:1939] T 65f678544c88419cbe820d11dda2b2f4 P f1478e5cb49148e4994ac79873f7a0ed: Deleting consensus metadata
I20260928 08:37:58.913972 7935 tablet_replica.cc:330] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6: stopping tablet replica
I20260928 08:37:58.914201 7935 raft_consensus.cc:2229] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:37:58.914304 7933 ts_tablet_manager.cc:1918] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:37:58.914441 7933 log.cc:1196] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/65f678544c88419cbe820d11dda2b2f4
I20260928 08:37:58.914849 7935 raft_consensus.cc:2258] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:37:58.914906 7933 ts_tablet_manager.cc:1939] T 65f678544c88419cbe820d11dda2b2f4 P 5143bb66a241433383f2648dcb812a64: Deleting consensus metadata
I20260928 08:37:58.915454 7139 catalog_manager.cc:4841] TS f1478e5cb49148e4994ac79873f7a0ed (127.0.97.1:44565): tablet 65f678544c88419cbe820d11dda2b2f4 (table test-table [id=8e0caeaa3eb34ed4beefc55e3f7a4efc]) successfully deleted
I20260928 08:37:58.915922 7142 catalog_manager.cc:4841] TS 5143bb66a241433383f2648dcb812a64 (127.0.97.2:35875): tablet 65f678544c88419cbe820d11dda2b2f4 (table test-table [id=8e0caeaa3eb34ed4beefc55e3f7a4efc]) successfully deleted
I20260928 08:37:58.917543 7935 ts_tablet_manager.cc:1905] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:37:58.925369 7935 ts_tablet_manager.cc:1918] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:37:58.925547 7935 log.cc:1196] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/65f678544c88419cbe820d11dda2b2f4
I20260928 08:37:58.926066 7935 ts_tablet_manager.cc:1939] T 65f678544c88419cbe820d11dda2b2f4 P 3b24f682e35048979dadd681aaf430c6: Deleting consensus metadata
I20260928 08:37:58.927080 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:58.898067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.0.97.1:44565)
0928 08:37:58.898288 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:58.898297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:58.908200 (+ 9903us) server_negotiation.cc:407] Connection header received
0928 08:37:58.908615 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:58.908621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:58.908716 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:58.908795 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:58.910707 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.911842 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.915117 (+ 3275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:58.915724 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:58.915835 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:58.919720 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:58.919738 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:58.919758 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:58.919796 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:58.923691 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:58.924714 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:58.924721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:58.924724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:58.924837 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:58.925305 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:58.925309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:58.925312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:58.925897 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:58.926270 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:58.926497 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 08:37:58.926730 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 08:37:58.927472 7140 catalog_manager.cc:4841] TS 3b24f682e35048979dadd681aaf430c6 (127.0.97.3:40031): tablet 65f678544c88419cbe820d11dda2b2f4 (table test-table [id=8e0caeaa3eb34ed4beefc55e3f7a4efc]) successfully deleted
I20260928 08:37:58.927925 7516 generic_service.cc:196] Checking for leaks (request via RPC)
W20260928 08:37:58.927893 7140 catalog_manager.cc:4808] TS 3b24f682e35048979dadd681aaf430c6 (127.0.97.3:40031): delete failed for tablet 65f678544c88419cbe820d11dda2b2f4 because the tablet was not found. No further retry: Not found: Tablet not found: 65f678544c88419cbe820d11dda2b2f4
I20260928 08:37:59.061110 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 7324
I20260928 08:37:59.083213 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 7533
I20260928 08:37:59.098394 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:59.084044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47392 (local address 127.0.97.2:35875)
0928 08:37:59.084215 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:59.084219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:59.084237 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:37:59.084470 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:59.084476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:59.084542 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:59.084625 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:59.086328 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:59.087201 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:59.089043 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:59.089414 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:59.089514 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:59.092640 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:59.092652 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:59.092668 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:59.092708 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:59.095576 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:59.096324 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:59.096332 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:59.096335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:59.096477 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:59.096846 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:59.096849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:59.096851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:59.097235 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:59.097468 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:59.097773 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 08:37:59.097954 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:37:59.099166 7664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:59.110307 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:59.109844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.0.1:57786)
0928 08:37:59.110023 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:59.110136 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 08:37:59.111608 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:59.111058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42403 (local address 127.0.0.1:57796)
0928 08:37:59.111315 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 08:37:59.111407 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260928 08:37:59.202839 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 7533
I20260928 08:37:59.221450 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 7682
I20260928 08:37:59.237650 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:37:59.222227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56986 (local address 127.0.97.3:40031)
0928 08:37:59.222421 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 08:37:59.222425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:37:59.222574 (+ 149us) server_negotiation.cc:407] Connection header received
0928 08:37:59.222775 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:37:59.222779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:37:59.222844 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:37:59.222928 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:37:59.224776 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:59.225531 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:59.227338 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:37:59.227765 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:37:59.227862 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:37:59.231479 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:37:59.231495 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:37:59.231521 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:37:59.231554 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:37:59.234778 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:59.235604 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:59.235611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:59.235614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:59.235773 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:37:59.236220 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:37:59.236223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:37:59.236225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:37:59.236559 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:37:59.236807 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 08:37:59.237139 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 08:37:59.237315 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:37:59.238456 7806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:59.369104 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 7682
I20260928 08:37:59.390859 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 7016
I20260928 08:37:59.391933 7185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:59.512825 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 7016
W20260928 08:37:59.534010 7328 connection.cc:535] server connection from 127.0.0.1:57092 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 08:37:59.534380 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 7119
I20260928 08:37:59.535441 7372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:37:59.649443 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 7119
2026-09-28T08:37:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9806 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:37:59 dist-test-slave-w6c0 krb5kdc[7944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:37:59 dist-test-slave-w6c0 krb5kdc[7944](info): set up 2 sockets
Sep 28 08:37:59 dist-test-slave-w6c0 krb5kdc[7944](info): commencing operation
krb5kdc: starting...
W20260928 08:38:00.751507 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:38:00 dist-test-slave-w6c0 krb5kdc[7944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:38:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:38:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260928 08:38:00.963043 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46243
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:38:01.285941 7957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:01.286217 7957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:01.286391 7957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:01.293929 7957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:38:01.294028 7957 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 08:38:01.294123 7957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:01.294237 7957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:01.294303 7957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:38:01.294391 7957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:38:01.304261 7957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46243
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:01.305502 7957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:01.306846 7957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:01.318164 7962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:01.318265 7965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:01.318282 7963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:01.318893 7957 server_base.cc:1034] running on GCE node
I20260928 08:38:01.319608 7957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:01.320881 7957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:01.322129 7957 hybrid_clock.cc:648] HybridClock initialized: now 1790584681322081 us; error 44 us; skew 500 ppm
I20260928 08:38:01.322454 7957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:38:01 dist-test-slave-w6c0 krb5kdc[7944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584681, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:38:01.325469 7957 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260928 08:38:01.326992 7957 webserver.cc:460] Webserver started at http://127.0.97.62:35999/ using document root <none> and password file <none>
I20260928 08:38:01.327700 7957 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:01.327831 7957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:01.328159 7957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:01.331120 7957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "183de3d8e93544c4a2c86d01a29dae37"
format_stamp: "Formatted at 2026-09-28 08:38:01 on dist-test-slave-w6c0"
server_key: "a31bae504c5b75675246f8886f7a2b37"
server_key_iv: "6898b0a510563f78faee10609fc5da55"
server_key_version: "encryptionkey@0"
I20260928 08:38:01.331925 7957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "183de3d8e93544c4a2c86d01a29dae37"
format_stamp: "Formatted at 2026-09-28 08:38:01 on dist-test-slave-w6c0"
server_key: "a31bae504c5b75675246f8886f7a2b37"
server_key_iv: "6898b0a510563f78faee10609fc5da55"
server_key_version: "encryptionkey@0"
I20260928 08:38:01.336804 7957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:38:01.340116 7972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:01.341311 7957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 08:38:01.341488 7957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "183de3d8e93544c4a2c86d01a29dae37"
format_stamp: "Formatted at 2026-09-28 08:38:01 on dist-test-slave-w6c0"
server_key: "a31bae504c5b75675246f8886f7a2b37"
server_key_iv: "6898b0a510563f78faee10609fc5da55"
server_key_version: "encryptionkey@0"
I20260928 08:38:01.341697 7957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:01.354786 7957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:01.358426 7957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:01.358700 7957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:01.381654 7957 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46243
I20260928 08:38:01.381672 8033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46243 every 8 connection(s)
I20260928 08:38:01.383147 7957 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:38:01.386719 8034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:01.392957 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 7957
I20260928 08:38:01.393177 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:38:01.394006 388 external_mini_cluster.cc:1428] Setting key 8931847a66715f4d786cd2a24550011d
I20260928 08:38:01.397656 8034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37: Bootstrap starting.
I20260928 08:38:01.401026 8034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:01.402139 8034 log.cc:824] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37: Log is configured to *not* fsync() on all Append() calls
I20260928 08:38:01.405180 8034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37: No bootstrap required, opened a new log
Sep 28 08:38:01 dist-test-slave-w6c0 krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:38:01.415891 8034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183de3d8e93544c4a2c86d01a29dae37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46243 } }
I20260928 08:38:01.416371 8034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:01.416488 8034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183de3d8e93544c4a2c86d01a29dae37, State: Initialized, Role: FOLLOWER
I20260928 08:38:01.417186 8034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183de3d8e93544c4a2c86d01a29dae37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46243 } }
I20260928 08:38:01.417511 8034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:38:01.417662 8034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:38:01.417850 8034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:01.418272 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:01.396116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53614 (local address 127.0.97.62:46243)
0928 08:38:01.396952 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:01.396963 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:01.397005 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:38:01.398973 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:01.398998 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:01.399394 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:01.399716 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:01.401470 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:01.403045 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:01.404324 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:01.405411 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:01.405605 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:01.409351 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:01.409369 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:01.409401 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:01.409452 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:01.413687 (+ 4235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:01.414332 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:01.414342 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:01.414357 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:01.414611 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:01.414959 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:01.414965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:01.414969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:01.415689 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:01.416019 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:01.416332 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 08:38:01.416795 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":412,"threads_started":1}
I20260928 08:38:01.421052 8034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183de3d8e93544c4a2c86d01a29dae37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46243 } }
I20260928 08:38:01.421559 8034 leader_election.cc:302] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 183de3d8e93544c4a2c86d01a29dae37; no voters:
I20260928 08:38:01.422067 8034 leader_election.cc:288] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:38:01.422165 8039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:38:01.422938 8039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37 [term 1 LEADER]: Becoming Leader. State: Replica: 183de3d8e93544c4a2c86d01a29dae37, State: Running, Role: LEADER
I20260928 08:38:01.423504 8039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183de3d8e93544c4a2c86d01a29dae37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46243 } }
I20260928 08:38:01.424331 8034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:38:01.425985 8040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "183de3d8e93544c4a2c86d01a29dae37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183de3d8e93544c4a2c86d01a29dae37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46243 } } }
I20260928 08:38:01.426378 8040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37 [sys.catalog]: This master's current role is: LEADER
I20260928 08:38:01.426944 8041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 183de3d8e93544c4a2c86d01a29dae37. Latest consensus state: current_term: 1 leader_uuid: "183de3d8e93544c4a2c86d01a29dae37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183de3d8e93544c4a2c86d01a29dae37" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46243 } } }
I20260928 08:38:01.427212 8041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37 [sys.catalog]: This master's current role is: LEADER
I20260928 08:38:01.428833 8046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:38:01.433465 8046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:38:01.441761 8046 catalog_manager.cc:1347] Generated new cluster ID: 676d5886f25742dcb018aa1eeae36c4d
I20260928 08:38:01.441865 8046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:38:01.454957 8046 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:38:01.456885 8046 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:38:01.469054 8046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37: Generated new TSK 0
I20260928 08:38:01.470166 8046 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260928 08:38:01.524780 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46243
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:38:01.846043 8060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:01.846369 8060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:01.846562 8060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:01.853685 8060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:01.853832 8060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:01.853983 8060 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:38:01.854130 8060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:38:01.863785 8060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:01.864988 8060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:01.866243 8060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:01.878780 8066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:01.878834 8068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:01.878857 8065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:01.879963 8060 server_base.cc:1034] running on GCE node
I20260928 08:38:01.880549 8060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:01.881378 8060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:01.882612 8060 hybrid_clock.cc:648] HybridClock initialized: now 1790584681882552 us; error 34 us; skew 500 ppm
I20260928 08:38:01.882923 8060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:38:01 dist-test-slave-w6c0 krb5kdc[7944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584681, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:38:01.886657 8060 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260928 08:38:01.888067 8060 webserver.cc:460] Webserver started at http://127.0.97.1:41619/ using document root <none> and password file <none>
I20260928 08:38:01.888756 8060 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:01.888840 8060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:01.889103 8060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:01.892055 8060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "ad3bbb2e240747db8627864f727715cf"
format_stamp: "Formatted at 2026-09-28 08:38:01 on dist-test-slave-w6c0"
server_key: "cbbd301eac92c34d0673d35b2d280c00"
server_key_iv: "524cbc74d508126440c83e2df64bd7cb"
server_key_version: "encryptionkey@0"
I20260928 08:38:01.892814 8060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ad3bbb2e240747db8627864f727715cf"
format_stamp: "Formatted at 2026-09-28 08:38:01 on dist-test-slave-w6c0"
server_key: "cbbd301eac92c34d0673d35b2d280c00"
server_key_iv: "524cbc74d508126440c83e2df64bd7cb"
server_key_version: "encryptionkey@0"
I20260928 08:38:01.897660 8060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:38:01.900876 8075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:01.902097 8060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:38:01.902261 8060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "ad3bbb2e240747db8627864f727715cf"
format_stamp: "Formatted at 2026-09-28 08:38:01 on dist-test-slave-w6c0"
server_key: "cbbd301eac92c34d0673d35b2d280c00"
server_key_iv: "524cbc74d508126440c83e2df64bd7cb"
server_key_version: "encryptionkey@0"
I20260928 08:38:01.902443 8060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:01.928915 8060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:01.933171 8060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:01.933447 8060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:01.936148 8060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:38:01.936224 8060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:01.936407 8060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:38:01.936488 8060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:38:01 dist-test-slave-w6c0 krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584681, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:38:01.961174 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:01.936609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38807 (local address 127.0.97.62:46243)
0928 08:38:01.938887 (+ 2278us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:01.938896 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:01.939819 (+ 923us) server_negotiation.cc:407] Connection header received
0928 08:38:01.942259 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:01.942267 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:01.942378 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:01.942523 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:01.944644 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:01.946058 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:01.949005 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:01.949639 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:01.949784 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:01.954969 (+ 5185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:01.954973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:01.954976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:01.955006 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:01.957432 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:01.958119 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:01.958128 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:01.958131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:01.958239 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:01.958742 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:01.958747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:01.958750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:01.959149 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:01.959309 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:01.960308 (+ 999us) server_negotiation.cc:299] Negotiation successful
0928 08:38:01.960614 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2132,"thread_start_us":2040,"threads_started":1}
I20260928 08:38:01.961627 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:01.938186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46243 (local address 127.0.97.1:38807)
0928 08:38:01.939369 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:01.939427 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:01.941860 (+ 2433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:01.942848 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:01.942859 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:01.943609 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:01.944432 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:01.944465 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:01.946259 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:01.946271 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:01.948827 (+ 2556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:01.948842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:01.949956 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:01.949966 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:01.950330 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:01.951403 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:01.951425 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:01.954737 (+ 3312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:38:01.957568 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:01.957577 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:01.957592 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:01.957993 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:01.958358 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:01.958363 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:01.958367 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:01.958628 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:01.959378 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:38:01.959384 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:38:01.959891 (+ 507us) client_negotiation.cc:769] Sending connection context
0928 08:38:01.960127 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 08:38:01.960494 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"thread_start_us":585,"threads_started":1}
I20260928 08:38:01.985991 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:01.967462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46243 (local address 127.0.97.1:59557)
0928 08:38:01.968444 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:01.968471 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:01.968663 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:01.969108 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:01.969116 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:01.970049 (+ 933us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:01.970610 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:01.970632 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:01.972138 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:01.972146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:01.974458 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:01.974478 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:01.975763 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:01.975772 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:01.975949 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:01.976723 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:01.976762 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:01.978029 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:38:01.982269 (+ 4240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:01.982275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:01.982279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:01.982898 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:01.983432 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:01.983439 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:01.983447 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:01.983620 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:01.984420 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:38:01.984427 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:38:01.984630 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 08:38:01.984795 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 08:38:01.985200 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":642,"threads_started":1}
I20260928 08:38:01.986308 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:01.967233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59557 (local address 127.0.97.62:46243)
0928 08:38:01.967753 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:01.967762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:01.968560 (+ 798us) server_negotiation.cc:407] Connection header received
0928 08:38:01.968845 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:01.968853 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:01.968939 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:01.969042 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:01.970789 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:01.971979 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:01.974623 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:01.975343 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:01.975486 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:01.978271 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:01.978279 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:01.978285 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:01.978339 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:01.982103 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:01.983064 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:01.983071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:01.983076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:01.983241 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:01.983775 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:01.983782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:01.983786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:01.984174 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:01.984332 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:01.985692 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0928 08:38:01.985939 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359}
I20260928 08:38:01.990110 8060 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44399
I20260928 08:38:01.990130 8195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44399 every 8 connection(s)
I20260928 08:38:01.991459 8060 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:38:01.993330 8196 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46243
I20260928 08:38:01.993566 8196 heartbeater.cc:464] Registering TS with master...
I20260928 08:38:01.994203 8196 heartbeater.cc:511] Master 127.0.97.62:46243 requested a full tablet report, sending...
I20260928 08:38:01.996078 7989 ts_manager.cc:194] Registered new tserver with Master: ad3bbb2e240747db8627864f727715cf (127.0.97.1:44399)
I20260928 08:38:01.997499 7989 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38807
I20260928 08:38:02.000149 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 8060
I20260928 08:38:02.000344 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:38:02.000962 388 external_mini_cluster.cc:1428] Setting key e1971a3486b8e9672c59f9710702262a
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260928 08:38:02.050370 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46243
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:38:02.371974 8200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:02.372268 8200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:02.372424 8200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:02.379809 8200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:02.379912 8200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:02.380110 8200 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:38:02.380252 8200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:38:02.390415 8200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:02.391700 8200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:02.393105 8200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:02.405946 8208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:02.405908 8206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:02.405992 8205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:02.406888 8200 server_base.cc:1034] running on GCE node
I20260928 08:38:02.407485 8200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:02.408381 8200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:02.409597 8200 hybrid_clock.cc:648] HybridClock initialized: now 1790584682409529 us; error 38 us; skew 500 ppm
I20260928 08:38:02.409845 8200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:38:02 dist-test-slave-w6c0 krb5kdc[7944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584682, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:38:02.413012 8200 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260928 08:38:02.414418 8200 webserver.cc:460] Webserver started at http://127.0.97.2:34829/ using document root <none> and password file <none>
I20260928 08:38:02.415119 8200 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:02.415239 8200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:02.415565 8200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:02.418329 8200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "3ef0262ddf5a43339816d2615819f557"
format_stamp: "Formatted at 2026-09-28 08:38:02 on dist-test-slave-w6c0"
server_key: "b7533228aa9ac52e658d947f074db530"
server_key_iv: "46251796bd9b6221d19617e57c0e55d0"
server_key_version: "encryptionkey@0"
I20260928 08:38:02.419179 8200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3ef0262ddf5a43339816d2615819f557"
format_stamp: "Formatted at 2026-09-28 08:38:02 on dist-test-slave-w6c0"
server_key: "b7533228aa9ac52e658d947f074db530"
server_key_iv: "46251796bd9b6221d19617e57c0e55d0"
server_key_version: "encryptionkey@0"
I20260928 08:38:02.424067 8200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:38:02.427517 8215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:02.428995 8200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:38:02.429198 8200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "3ef0262ddf5a43339816d2615819f557"
format_stamp: "Formatted at 2026-09-28 08:38:02 on dist-test-slave-w6c0"
server_key: "b7533228aa9ac52e658d947f074db530"
server_key_iv: "46251796bd9b6221d19617e57c0e55d0"
server_key_version: "encryptionkey@0"
I20260928 08:38:02.429414 8200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:02.443892 8200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:02.448443 8200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:02.448711 8200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:02.451417 8200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:38:02.451543 8200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:02.451756 8200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:38:02.451848 8200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:38:02 dist-test-slave-w6c0 krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584682, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:38:02.477106 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:02.451831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59627 (local address 127.0.97.62:46243)
0928 08:38:02.452016 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:02.452021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:02.453453 (+ 1432us) server_negotiation.cc:407] Connection header received
0928 08:38:02.455678 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:02.455683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:02.455775 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:02.455887 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:02.458034 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:02.459189 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:02.462544 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:02.463356 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:02.463483 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:02.468635 (+ 5152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:02.468645 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:02.468652 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:02.468711 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:02.472832 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:02.473890 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:02.473896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:02.473901 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:02.474055 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:02.474547 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:02.474560 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:02.474563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:02.474982 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:02.475121 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:02.476273 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0928 08:38:02.476574 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 08:38:02.477617 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:02.452416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46243 (local address 127.0.97.2:59627)
0928 08:38:02.453287 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:02.453322 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:02.455353 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:02.456129 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:02.456136 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:02.456964 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:02.457856 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:02.457875 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:02.459414 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:02.459422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:02.462266 (+ 2844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:02.462281 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:02.463904 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:02.463925 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:02.464287 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:02.465129 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:02.465175 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:02.468389 (+ 3214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:38:02.473019 (+ 4630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:02.473033 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:02.473050 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:02.473653 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:02.474177 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:02.474182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:02.474186 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:02.474399 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:02.475203 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:38:02.475211 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:38:02.475713 (+ 502us) client_negotiation.cc:769] Sending connection context
0928 08:38:02.476107 (+ 394us) client_negotiation.cc:240] Negotiation successful
0928 08:38:02.476506 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":409,"threads_started":1}
I20260928 08:38:02.498337 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:02.482976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46205 (local address 127.0.97.62:46243)
0928 08:38:02.483200 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:02.483209 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:02.483934 (+ 725us) server_negotiation.cc:407] Connection header received
0928 08:38:02.484059 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:02.484067 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:02.484164 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:02.484291 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:02.485167 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:02.486616 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:02.488706 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:02.489471 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:02.489609 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:02.492496 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:02.492503 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:02.492507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:02.492544 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:02.495442 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:02.496239 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:02.496246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:02.496249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:02.496371 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:02.496721 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:02.496726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:02.496730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:02.497214 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:02.497376 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:02.497755 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 08:38:02.497962 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 08:38:02.498412 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:02.483115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46243 (local address 127.0.97.2:46205)
0928 08:38:02.483827 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:02.483842 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:02.483945 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:02.484320 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:02.484323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:02.484760 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:02.485031 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:02.485040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:02.486777 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:02.486787 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:02.488517 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:02.488532 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:02.489884 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:02.489891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:02.490203 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:02.490999 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:02.491034 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:02.492315 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:38:02.495647 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:02.495651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:02.495657 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:02.496105 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:02.496481 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:02.496486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:02.496490 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:02.496612 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:02.497347 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:38:02.497352 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:38:02.497524 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 08:38:02.497670 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 08:38:02.497972 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":425,"threads_started":1}
I20260928 08:38:02.508785 8200 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44931
I20260928 08:38:02.508806 8334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44931 every 8 connection(s)
I20260928 08:38:02.509964 8200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:38:02.510789 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 8200
I20260928 08:38:02.510948 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:38:02.511533 388 external_mini_cluster.cc:1428] Setting key 9d79180280b0ef044fa7be552d679f1a
I20260928 08:38:02.512100 8335 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46243
I20260928 08:38:02.512342 8335 heartbeater.cc:464] Registering TS with master...
I20260928 08:38:02.513127 8335 heartbeater.cc:511] Master 127.0.97.62:46243 requested a full tablet report, sending...
I20260928 08:38:02.514468 7989 ts_manager.cc:194] Registered new tserver with Master: 3ef0262ddf5a43339816d2615819f557 (127.0.97.2:44931)
I20260928 08:38:02.515496 7989 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59627
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260928 08:38:02.558563 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46243
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:38:02.884231 8339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:02.884513 8339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:02.884666 8339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:02.892351 8339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:02.892473 8339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:02.892643 8339 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:38:02.892813 8339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:38:02.902869 8339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:02.904186 8339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:02.905478 8339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:02.919416 8344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:02.919436 8347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:02.919397 8345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:02.920392 8339 server_base.cc:1034] running on GCE node
I20260928 08:38:02.920984 8339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:02.921841 8339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:02.923074 8339 hybrid_clock.cc:648] HybridClock initialized: now 1790584682923010 us; error 36 us; skew 500 ppm
I20260928 08:38:02.923367 8339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:38:02 dist-test-slave-w6c0 krb5kdc[7944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584682, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:38:02.932827 8339 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260928 08:38:02.934453 8339 webserver.cc:460] Webserver started at http://127.0.97.3:46161/ using document root <none> and password file <none>
I20260928 08:38:02.935197 8339 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:02.935330 8339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:02.935668 8339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:02.938694 8339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "304d64805d4241d3914323de7627a8d7"
format_stamp: "Formatted at 2026-09-28 08:38:02 on dist-test-slave-w6c0"
server_key: "4ac2a3403fb3edb0a7a9520fd8671303"
server_key_iv: "d31b3d79ede1d7bd2ebdd22b9804e92a"
server_key_version: "encryptionkey@0"
I20260928 08:38:02.939496 8339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "304d64805d4241d3914323de7627a8d7"
format_stamp: "Formatted at 2026-09-28 08:38:02 on dist-test-slave-w6c0"
server_key: "4ac2a3403fb3edb0a7a9520fd8671303"
server_key_iv: "d31b3d79ede1d7bd2ebdd22b9804e92a"
server_key_version: "encryptionkey@0"
I20260928 08:38:02.944592 8339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260928 08:38:02.948000 8354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:02.949394 8339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:38:02.949590 8339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "304d64805d4241d3914323de7627a8d7"
format_stamp: "Formatted at 2026-09-28 08:38:02 on dist-test-slave-w6c0"
server_key: "4ac2a3403fb3edb0a7a9520fd8671303"
server_key_iv: "d31b3d79ede1d7bd2ebdd22b9804e92a"
server_key_version: "encryptionkey@0"
I20260928 08:38:02.949816 8339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:02.968070 8339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:02.973183 8339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:02.973524 8339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:02.977797 8339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:38:02.978009 8339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 08:38:02.978209 8339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:38:02.978333 8339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:38:02 dist-test-slave-w6c0 krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584682, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:38:03.000002 8196 heartbeater.cc:503] Master 127.0.97.62:46243 was elected leader, sending a full tablet report...
I20260928 08:38:03.009016 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:02.978357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46165 (local address 127.0.97.62:46243)
0928 08:38:02.978619 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:02.978626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:02.981594 (+ 2968us) server_negotiation.cc:407] Connection header received
0928 08:38:02.986623 (+ 5029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:02.986633 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:02.986761 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:02.986935 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:02.989586 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:02.990821 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:02.993367 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:02.994164 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:02.994282 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:02.999834 (+ 5552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:02.999840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:02.999845 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:02.999886 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:03.004155 (+ 4269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:03.005194 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:03.005202 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:03.005205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:03.005348 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:03.006014 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:03.006022 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:03.006026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:03.006358 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:03.006514 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:03.008156 (+ 1642us) server_negotiation.cc:299] Negotiation successful
0928 08:38:03.008490 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 08:38:03.009805 8363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:02.979174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46243 (local address 127.0.97.3:46165)
0928 08:38:02.980610 (+ 1436us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:02.981443 (+ 833us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:02.984936 (+ 3493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:02.987627 (+ 2691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:02.987643 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:02.988476 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:02.989336 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:02.989367 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:02.991039 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:02.991044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:02.993180 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:02.993196 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:02.995000 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:02.995007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:02.995272 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:02.996202 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:02.996231 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:02.999528 (+ 3297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:38:03.004328 (+ 4800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:03.004343 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:03.004364 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:03.005069 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:03.005528 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:03.005534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:03.005539 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:03.005879 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:03.006542 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:38:03.006554 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:38:03.007095 (+ 541us) client_negotiation.cc:769] Sending connection context
0928 08:38:03.007426 (+ 331us) client_negotiation.cc:240] Negotiation successful
0928 08:38:03.008091 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"thread_start_us":814,"threads_started":1}
I20260928 08:38:03.035175 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.017127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46243 (local address 127.0.97.3:55839)
0928 08:38:03.017844 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:03.017860 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:03.018040 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:03.018665 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:03.018669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:03.019479 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:03.020063 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.020080 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.021635 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.021640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.023401 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.023422 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.024416 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.024422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.024700 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.025516 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:03.025548 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:03.027040 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:38:03.031496 (+ 4456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:03.031503 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:03.031512 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:03.032270 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:03.032806 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:03.032811 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:03.032815 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:03.033007 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:03.033785 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:38:03.033793 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:38:03.034052 (+ 259us) client_negotiation.cc:769] Sending connection context
0928 08:38:03.034190 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 08:38:03.034667 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":434,"threads_started":1}
I20260928 08:38:03.036544 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.016997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55839 (local address 127.0.97.62:46243)
0928 08:38:03.017168 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:03.017172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:03.017958 (+ 786us) server_negotiation.cc:407] Connection header received
0928 08:38:03.018202 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:03.018214 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:03.018337 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:03.018444 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:03.020257 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.021492 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.023616 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.024211 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.024361 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.027257 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:03.027265 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:03.027272 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:03.027317 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:03.031352 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:03.032411 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:03.032418 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:03.032424 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:03.032666 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:03.033208 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:03.033214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:03.033219 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:03.033551 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:03.035673 (+ 2122us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:03.035849 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 08:38:03.036126 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:38:03.047575 8339 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43111
I20260928 08:38:03.047607 8473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43111 every 8 connection(s)
I20260928 08:38:03.049635 8339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:38:03.050346 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 8339
I20260928 08:38:03.050558 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:38:03.051254 388 external_mini_cluster.cc:1428] Setting key 60e8896a1599c79a8d837825f24d3929
I20260928 08:38:03.052155 8474 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46243
I20260928 08:38:03.052449 8474 heartbeater.cc:464] Registering TS with master...
I20260928 08:38:03.053572 8474 heartbeater.cc:511] Master 127.0.97.62:46243 requested a full tablet report, sending...
I20260928 08:38:03.055451 7989 ts_manager.cc:194] Registered new tserver with Master: 304d64805d4241d3914323de7627a8d7 (127.0.97.3:43111)
I20260928 08:38:03.056860 7989 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46165
I20260928 08:38:03.067607 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:38:03 dist-test-slave-w6c0 krb5kdc[7944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584683, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:38:03 dist-test-slave-w6c0 krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:03 dist-test-slave-w6c0 krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:38:03.117754 8084 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:03.107193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53630 (local address 127.0.97.62:46243)
0928 08:38:03.107421 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:03.107429 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:03.107604 (+ 175us) server_negotiation.cc:407] Connection header received
0928 08:38:03.107860 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:03.107864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:03.107929 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:03.108039 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:03.109307 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.110048 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.111420 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.111855 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.111990 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.117435 (+ 5445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53630: BlockingRecv error: recv got EOF from 127.0.0.1:53630 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 08:38:03 dist-test-slave-w6c0 krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584683, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:38:03.133497 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.120566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53646 (local address 127.0.97.62:46243)
0928 08:38:03.120737 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:03.120741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:03.121002 (+ 261us) server_negotiation.cc:407] Connection header received
0928 08:38:03.121126 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:03.121130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:03.121208 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:03.121288 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:03.122392 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.123204 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.124387 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.124834 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.124925 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.128601 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:03.128605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:03.128609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:03.128637 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:03.131111 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:03.131755 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:03.131758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:03.131761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:03.131895 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:03.132242 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:03.132246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:03.132249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:03.132550 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:03.132761 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:03.133063 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 08:38:03.133274 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:38:03.137394 7989 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:38:03.144232 7989 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:38:03.179342 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.156669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44166 (local address 127.0.97.3:43111)
0928 08:38:03.169277 (+ 12608us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:03.169289 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:03.170040 (+ 751us) server_negotiation.cc:407] Connection header received
0928 08:38:03.170439 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:03.170448 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:03.170753 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:03.170957 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:38:03.172158 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.173924 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.176550 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.177790 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.177917 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.178167 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:03.178784 (+ 617us) server_negotiation.cc:299] Negotiation successful
0928 08:38:03.178952 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12405,"thread_start_us":12164,"threads_started":1}
I20260928 08:38:03.179391 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.156408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43111 (local address 127.0.0.1:44166)
0928 08:38:03.168614 (+ 12206us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:03.169382 (+ 768us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:03.169910 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:03.171280 (+ 1370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:03.171304 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:03.171334 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:38:03.171930 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.171940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.174136 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.174141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.176389 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.176403 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.178306 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.178315 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.178501 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.178573 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 08:38:03.178709 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 08:38:03.178907 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11840,"mutex_wait_us":137,"thread_start_us":13806,"threads_started":1}
I20260928 08:38:03.180080 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.156899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55786 (local address 127.0.97.1:44399)
0928 08:38:03.167630 (+ 10731us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:03.167645 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:03.170337 (+ 2692us) server_negotiation.cc:407] Connection header received
0928 08:38:03.170696 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:03.170706 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:03.170916 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:03.171074 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:38:03.172375 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.173924 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.176736 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.178650 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.178809 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.178994 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:03.179714 (+ 720us) server_negotiation.cc:299] Negotiation successful
0928 08:38:03.179882 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10492,"thread_start_us":10745,"threads_started":1}
I20260928 08:38:03.180846 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.156636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44399 (local address 127.0.0.1:55786)
0928 08:38:03.170201 (+ 13565us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:03.170223 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:03.170473 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:03.171339 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:03.171344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:03.171367 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:38:03.172187 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.172195 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.174603 (+ 2408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.174610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.176536 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.176551 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.179170 (+ 2619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.179176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.179351 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.179397 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 08:38:03.179496 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 08:38:03.180375 (+ 879us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13380,"mutex_wait_us":105,"thread_start_us":11450,"threads_started":1}
I20260928 08:38:03.181257 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.159368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44931 (local address 127.0.0.1:47460)
0928 08:38:03.170472 (+ 11104us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:03.170488 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:03.170607 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:03.171356 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:03.171359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:03.171375 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:38:03.172182 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.172195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.175740 (+ 3545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.175745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.178268 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.178287 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.180363 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.180369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.180550 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.180583 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 08:38:03.180683 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 08:38:03.180833 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10997,"mutex_wait_us":63,"thread_start_us":10440,"threads_started":1}
I20260928 08:38:03.181689 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.159931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47460 (local address 127.0.97.2:44931)
0928 08:38:03.169593 (+ 9662us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:03.169605 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:03.170735 (+ 1130us) server_negotiation.cc:407] Connection header received
0928 08:38:03.170825 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:03.170831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:03.171012 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:03.171165 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:38:03.174131 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.175598 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.178444 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.180041 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.180165 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.180572 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:03.180908 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 08:38:03.181066 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9482,"thread_start_us":9339,"threads_started":1}
I20260928 08:38:03.183791 8409 tablet_service.cc:1467] Processing CreateTablet for tablet dbf5d27020e14724a0e17673d86496b2 (DEFAULT_TABLE table=test-table [id=ecf6a759362a42d2b482055404b1c294]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:38:03.184695 8265 tablet_service.cc:1467] Processing CreateTablet for tablet dbf5d27020e14724a0e17673d86496b2 (DEFAULT_TABLE table=test-table [id=ecf6a759362a42d2b482055404b1c294]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:38:03.185386 8409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf5d27020e14724a0e17673d86496b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:03.185066 8126 tablet_service.cc:1467] Processing CreateTablet for tablet dbf5d27020e14724a0e17673d86496b2 (DEFAULT_TABLE table=test-table [id=ecf6a759362a42d2b482055404b1c294]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:38:03.185839 8265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf5d27020e14724a0e17673d86496b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:03.186272 8126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf5d27020e14724a0e17673d86496b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:03.203941 8500 tablet_bootstrap.cc:492] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7: Bootstrap starting.
I20260928 08:38:03.207196 8498 tablet_bootstrap.cc:492] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf: Bootstrap starting.
I20260928 08:38:03.210047 8498 tablet_bootstrap.cc:654] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:03.210049 8500 tablet_bootstrap.cc:654] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:03.210637 8499 tablet_bootstrap.cc:492] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557: Bootstrap starting.
I20260928 08:38:03.211236 8500 log.cc:824] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7: Log is configured to *not* fsync() on all Append() calls
I20260928 08:38:03.211315 8498 log.cc:824] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf: Log is configured to *not* fsync() on all Append() calls
I20260928 08:38:03.214004 8499 tablet_bootstrap.cc:654] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:03.214584 8500 tablet_bootstrap.cc:492] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7: No bootstrap required, opened a new log
I20260928 08:38:03.214586 8498 tablet_bootstrap.cc:492] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf: No bootstrap required, opened a new log
I20260928 08:38:03.214872 8498 ts_tablet_manager.cc:1397] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 08:38:03.214872 8500 ts_tablet_manager.cc:1397] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260928 08:38:03.215169 8499 log.cc:824] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557: Log is configured to *not* fsync() on all Append() calls
I20260928 08:38:03.218133 8499 tablet_bootstrap.cc:492] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557: No bootstrap required, opened a new log
I20260928 08:38:03.218402 8499 ts_tablet_manager.cc:1397] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260928 08:38:03.223096 8500 raft_consensus.cc:348] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } } peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } }
I20260928 08:38:03.223460 8500 raft_consensus.cc:374] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:03.223560 8500 raft_consensus.cc:729] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 304d64805d4241d3914323de7627a8d7, State: Initialized, Role: FOLLOWER
I20260928 08:38:03.224241 8500 consensus_queue.cc:260] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } } peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } }
I20260928 08:38:03.225464 8474 heartbeater.cc:503] Master 127.0.97.62:46243 was elected leader, sending a full tablet report...
I20260928 08:38:03.225279 8498 raft_consensus.cc:348] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } } peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } }
I20260928 08:38:03.225662 8498 raft_consensus.cc:374] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:03.225773 8498 raft_consensus.cc:729] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad3bbb2e240747db8627864f727715cf, State: Initialized, Role: FOLLOWER
I20260928 08:38:03.226051 8500 ts_tablet_manager.cc:1428] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260928 08:38:03.226361 8498 consensus_queue.cc:260] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } } peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } }
I20260928 08:38:03.228295 8498 ts_tablet_manager.cc:1428] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260928 08:38:03.229818 8499 raft_consensus.cc:348] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } } peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } }
I20260928 08:38:03.230257 8499 raft_consensus.cc:374] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:03.230374 8499 raft_consensus.cc:729] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef0262ddf5a43339816d2615819f557, State: Initialized, Role: FOLLOWER
I20260928 08:38:03.231209 8499 consensus_queue.cc:260] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } } peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } }
I20260928 08:38:03.232396 8506 raft_consensus.cc:482] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:38:03.232599 8335 heartbeater.cc:503] Master 127.0.97.62:46243 was elected leader, sending a full tablet report...
I20260928 08:38:03.232743 8506 raft_consensus.cc:504] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } } peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } }
I20260928 08:38:03.233103 8499 ts_tablet_manager.cc:1428] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260928 08:38:03.235033 8506 leader_election.cc:288] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 304d64805d4241d3914323de7627a8d7 (127.0.97.3:43111), ad3bbb2e240747db8627864f727715cf (127.0.97.1:44399)
I20260928 08:38:03.242247 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.235585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37135 (local address 127.0.97.1:44399)
0928 08:38:03.235769 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:03.235775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:03.235947 (+ 172us) server_negotiation.cc:407] Connection header received
0928 08:38:03.236262 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:03.236268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:03.236353 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:03.236418 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:38:03.237350 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.238272 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.240524 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.241301 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.241462 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.241566 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:03.241870 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 08:38:03.241995 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 08:38:03.242377 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.235776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45493 (local address 127.0.97.3:43111)
0928 08:38:03.235972 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:03.235978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:03.236580 (+ 602us) server_negotiation.cc:407] Connection header received
0928 08:38:03.236726 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:03.236734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:03.236808 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:03.236909 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:38:03.237586 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.238480 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.240675 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.241440 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.241594 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.241706 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:03.241985 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 08:38:03.242110 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 08:38:03.242475 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.235351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43111 (local address 127.0.97.2:45493)
0928 08:38:03.236469 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:03.236486 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:03.236605 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:03.236924 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:03.236928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:03.236955 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:38:03.237413 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.237426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.238658 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.238663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.240523 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.240534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.241657 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.241663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.241832 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.241854 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 08:38:03.241959 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 08:38:03.242080 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"mutex_wait_us":161,"thread_start_us":874,"threads_started":1}
I20260928 08:38:03.242811 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.235266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44399 (local address 127.0.97.2:37135)
0928 08:38:03.235825 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:03.235846 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:03.236076 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:03.236636 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:03.236641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:03.236668 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:38:03.237175 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.237187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.238409 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.238415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.240357 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.240369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.241518 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.241524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.241711 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.241735 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 08:38:03.241846 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 08:38:03.241975 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":71,"thread_start_us":291,"threads_started":1}
I20260928 08:38:03.243462 8429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf5d27020e14724a0e17673d86496b2" candidate_uuid: "3ef0262ddf5a43339816d2615819f557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "304d64805d4241d3914323de7627a8d7" is_pre_election: true
I20260928 08:38:03.243642 8151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf5d27020e14724a0e17673d86496b2" candidate_uuid: "3ef0262ddf5a43339816d2615819f557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad3bbb2e240747db8627864f727715cf" is_pre_election: true
I20260928 08:38:03.243934 8429 raft_consensus.cc:2454] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ef0262ddf5a43339816d2615819f557 in term 0.
I20260928 08:38:03.244197 8151 raft_consensus.cc:2454] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ef0262ddf5a43339816d2615819f557 in term 0.
I20260928 08:38:03.244601 8216 leader_election.cc:302] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 304d64805d4241d3914323de7627a8d7, 3ef0262ddf5a43339816d2615819f557; no voters:
W20260928 08:38:03.245052 8197 tablet.cc:2369] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:38:03.245333 8506 raft_consensus.cc:2781] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:38:03.245477 8506 raft_consensus.cc:482] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:38:03.245572 8506 raft_consensus.cc:3037] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:03.247913 8506 raft_consensus.cc:504] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } } peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } }
I20260928 08:38:03.248495 8506 leader_election.cc:288] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [CANDIDATE]: Term 1 election: Requested vote from peers 304d64805d4241d3914323de7627a8d7 (127.0.97.3:43111), ad3bbb2e240747db8627864f727715cf (127.0.97.1:44399)
I20260928 08:38:03.248919 8429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf5d27020e14724a0e17673d86496b2" candidate_uuid: "3ef0262ddf5a43339816d2615819f557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "304d64805d4241d3914323de7627a8d7"
I20260928 08:38:03.248986 8151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf5d27020e14724a0e17673d86496b2" candidate_uuid: "3ef0262ddf5a43339816d2615819f557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad3bbb2e240747db8627864f727715cf"
I20260928 08:38:03.249118 8429 raft_consensus.cc:3037] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:03.249243 8151 raft_consensus.cc:3037] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:03.251590 8429 raft_consensus.cc:2454] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ef0262ddf5a43339816d2615819f557 in term 1.
I20260928 08:38:03.251595 8151 raft_consensus.cc:2454] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ef0262ddf5a43339816d2615819f557 in term 1.
I20260928 08:38:03.251991 8216 leader_election.cc:302] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 304d64805d4241d3914323de7627a8d7, 3ef0262ddf5a43339816d2615819f557; no voters:
I20260928 08:38:03.252306 8506 raft_consensus.cc:2781] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:38:03.252485 8506 raft_consensus.cc:686] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [term 1 LEADER]: Becoming Leader. State: Replica: 3ef0262ddf5a43339816d2615819f557, State: Running, Role: LEADER
I20260928 08:38:03.252961 8506 consensus_queue.cc:237] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } } peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } }
I20260928 08:38:03.257731 7989 catalog_manager.cc:5495] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ef0262ddf5a43339816d2615819f557 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3ef0262ddf5a43339816d2615819f557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } health_report { overall_health: HEALTHY } } }
W20260928 08:38:03.261934 8336 tablet.cc:2369] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:38:03.289222 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.283839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47474 (local address 127.0.97.2:44931)
0928 08:38:03.284043 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:03.284048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:03.284067 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:38:03.284274 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:03.284277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:03.284354 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:03.284485 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:38:03.284815 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.285530 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.287042 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.287443 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.287560 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.287996 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:38:03.288562 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:38:03.288691 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:03.288934 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 08:38:03.289004 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 08:38:03.297406 8429 raft_consensus.cc:1261] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7 [term 1 FOLLOWER]: Refusing update from remote peer 3ef0262ddf5a43339816d2615819f557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:38:03.297406 8151 raft_consensus.cc:1261] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf [term 1 FOLLOWER]: Refusing update from remote peer 3ef0262ddf5a43339816d2615819f557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:38:03.298379 8507 consensus_queue.cc:1035] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:38:03.298859 8506 consensus_queue.cc:1035] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 08:38:03.306550 8475 tablet.cc:2369] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:38:03.315572 8514 mvcc.cc:205] Tried to move back new op lower bound from 7334234862776979456 to 7334234862609289216. Current Snapshot: MvccSnapshot[applied={T|T < 7334234862776979456}]
I20260928 08:38:03.324821 7988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:38:03.325189 7988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:38:03.328796 7988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37: Sending DeleteTablet for 3 replicas of tablet dbf5d27020e14724a0e17673d86496b2
I20260928 08:38:03.329859 8409 tablet_service.cc:1514] Processing DeleteTablet for tablet dbf5d27020e14724a0e17673d86496b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:38:03 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44166
I20260928 08:38:03.329908 8126 tablet_service.cc:1514] Processing DeleteTablet for tablet dbf5d27020e14724a0e17673d86496b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:38:03 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55786
I20260928 08:38:03.330159 8265 tablet_service.cc:1514] Processing DeleteTablet for tablet dbf5d27020e14724a0e17673d86496b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:38:03 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47460
I20260928 08:38:03.332209 7988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:38:03.333580 7988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:38:03.341104 8521 tablet_replica.cc:330] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7: stopping tablet replica
I20260928 08:38:03.341741 8521 raft_consensus.cc:2229] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:38:03.342348 8521 raft_consensus.cc:2258] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:38:03.343204 8523 tablet_replica.cc:330] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557: stopping tablet replica
I20260928 08:38:03.343873 8523 raft_consensus.cc:2229] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:38:03.344897 8125 tablet_service.cc:1467] Processing CreateTablet for tablet bdc9d1d181414941a62d72706d4111df (DEFAULT_TABLE table=test-table [id=0d6ef754cd8e4b53a906c13ea358e730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:38:03.345521 8125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc9d1d181414941a62d72706d4111df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:03.345474 8264 tablet_service.cc:1467] Processing CreateTablet for tablet bdc9d1d181414941a62d72706d4111df (DEFAULT_TABLE table=test-table [id=0d6ef754cd8e4b53a906c13ea358e730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:38:03.346005 8264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc9d1d181414941a62d72706d4111df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:03.344843 8523 raft_consensus.cc:2258] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:38:03.352551 8498 tablet_bootstrap.cc:492] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf: Bootstrap starting.
I20260928 08:38:03.352977 8409 tablet_service.cc:1467] Processing CreateTablet for tablet bdc9d1d181414941a62d72706d4111df (DEFAULT_TABLE table=test-table [id=0d6ef754cd8e4b53a906c13ea358e730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:38:03.353529 8409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc9d1d181414941a62d72706d4111df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:03.354313 8498 tablet_bootstrap.cc:654] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:03.360720 8523 ts_tablet_manager.cc:1905] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:38:03.363369 8522 tablet_replica.cc:330] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf: stopping tablet replica
I20260928 08:38:03.363582 8498 tablet_bootstrap.cc:492] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf: No bootstrap required, opened a new log
I20260928 08:38:03.363737 8498 ts_tablet_manager.cc:1397] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260928 08:38:03.363824 8522 raft_consensus.cc:2229] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:38:03.364372 8522 raft_consensus.cc:2258] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:38:03.364683 8498 raft_consensus.cc:348] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:03.365228 8498 raft_consensus.cc:374] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:03.365474 8499 tablet_bootstrap.cc:492] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557: Bootstrap starting.
I20260928 08:38:03.366586 8498 raft_consensus.cc:729] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad3bbb2e240747db8627864f727715cf, State: Initialized, Role: FOLLOWER
I20260928 08:38:03.366735 8522 ts_tablet_manager.cc:1905] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:38:03.366945 8498 consensus_queue.cc:260] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:03.367349 8499 tablet_bootstrap.cc:654] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:03.368733 8523 ts_tablet_manager.cc:1918] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:38:03.368877 8523 log.cc:1196] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/dbf5d27020e14724a0e17673d86496b2
I20260928 08:38:03.369361 8523 ts_tablet_manager.cc:1939] T dbf5d27020e14724a0e17673d86496b2 P 3ef0262ddf5a43339816d2615819f557: Deleting consensus metadata
I20260928 08:38:03.370275 8498 ts_tablet_manager.cc:1428] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260928 08:38:03.371073 7974 catalog_manager.cc:4841] TS 3ef0262ddf5a43339816d2615819f557 (127.0.97.2:44931): tablet dbf5d27020e14724a0e17673d86496b2 (table test-table [id=ecf6a759362a42d2b482055404b1c294]) successfully deleted
I20260928 08:38:03.372646 8521 ts_tablet_manager.cc:1905] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:38:03.373056 8522 ts_tablet_manager.cc:1918] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:38:03.373214 8522 log.cc:1196] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/wal/wals/dbf5d27020e14724a0e17673d86496b2
I20260928 08:38:03.373646 8522 ts_tablet_manager.cc:1939] T dbf5d27020e14724a0e17673d86496b2 P ad3bbb2e240747db8627864f727715cf: Deleting consensus metadata
I20260928 08:38:03.377169 8500 tablet_bootstrap.cc:492] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7: Bootstrap starting.
I20260928 08:38:03.378983 8500 tablet_bootstrap.cc:654] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:03.379467 8499 tablet_bootstrap.cc:492] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557: No bootstrap required, opened a new log
I20260928 08:38:03.379700 8499 ts_tablet_manager.cc:1397] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20260928 08:38:03.379853 8521 ts_tablet_manager.cc:1918] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:38:03.380067 8521 log.cc:1196] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/dbf5d27020e14724a0e17673d86496b2
I20260928 08:38:03.380769 8521 ts_tablet_manager.cc:1939] T dbf5d27020e14724a0e17673d86496b2 P 304d64805d4241d3914323de7627a8d7: Deleting consensus metadata
I20260928 08:38:03.382115 7976 catalog_manager.cc:4841] TS ad3bbb2e240747db8627864f727715cf (127.0.97.1:44399): tablet dbf5d27020e14724a0e17673d86496b2 (table test-table [id=ecf6a759362a42d2b482055404b1c294]) successfully deleted
I20260928 08:38:03.382815 7973 catalog_manager.cc:4841] TS 304d64805d4241d3914323de7627a8d7 (127.0.97.3:43111): tablet dbf5d27020e14724a0e17673d86496b2 (table test-table [id=ecf6a759362a42d2b482055404b1c294]) successfully deleted
I20260928 08:38:03.380918 8499 raft_consensus.cc:348] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:03.384167 8499 raft_consensus.cc:374] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:03.384336 8499 raft_consensus.cc:729] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef0262ddf5a43339816d2615819f557, State: Initialized, Role: FOLLOWER
I20260928 08:38:03.384651 8499 consensus_queue.cc:260] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:03.384913 8500 tablet_bootstrap.cc:492] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7: No bootstrap required, opened a new log
I20260928 08:38:03.385053 8500 ts_tablet_manager.cc:1397] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 08:38:03.385450 8499 ts_tablet_manager.cc:1428] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260928 08:38:03.385872 8500 raft_consensus.cc:348] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:03.386081 8500 raft_consensus.cc:374] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:03.386193 8500 raft_consensus.cc:729] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 304d64805d4241d3914323de7627a8d7, State: Initialized, Role: FOLLOWER
I20260928 08:38:03.386487 8500 consensus_queue.cc:260] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:03.388958 8500 ts_tablet_manager.cc:1428] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260928 08:38:03.558610 8504 raft_consensus.cc:482] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:38:03.558816 8504 raft_consensus.cc:504] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:03.560356 8504 leader_election.cc:288] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad3bbb2e240747db8627864f727715cf (127.0.97.1:44399), 3ef0262ddf5a43339816d2615819f557 (127.0.97.2:44931)
I20260928 08:38:03.568210 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.560747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52299 (local address 127.0.97.2:44931)
0928 08:38:03.560982 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:03.560992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:03.561724 (+ 732us) server_negotiation.cc:407] Connection header received
0928 08:38:03.561957 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:03.561961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:03.562058 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:03.562146 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:38:03.562805 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.563940 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.565871 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.567165 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.567332 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.567430 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:03.567767 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 08:38:03.567905 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:38:03.568419 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.560968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56437 (local address 127.0.97.1:44399)
0928 08:38:03.561145 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:03.561151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:03.561511 (+ 360us) server_negotiation.cc:407] Connection header received
0928 08:38:03.561812 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:03.561818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:03.561900 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:03.562007 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:38:03.562520 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.563653 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.565979 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.567222 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.567347 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.567435 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:03.568019 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 08:38:03.568144 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:38:03.568233 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.560673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44931 (local address 127.0.97.3:52299)
0928 08:38:03.561639 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:03.561655 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:03.561812 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:03.562196 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:03.562203 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:03.562234 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:38:03.562617 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.562628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.564061 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.564068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.565706 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.565718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.567362 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.567367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.567559 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.567605 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 08:38:03.567704 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 08:38:03.567852 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":24,"thread_start_us":462,"threads_started":1}
I20260928 08:38:03.568915 8505 raft_consensus.cc:482] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:38:03.569403 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.560584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44399 (local address 127.0.97.3:56437)
0928 08:38:03.561384 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:03.561405 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:03.561621 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:03.562033 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:03.562039 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:03.562065 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:38:03.562341 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.562353 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.563830 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.563835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.565848 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.565863 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.567598 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.567602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.567821 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.567854 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 08:38:03.568634 (+ 780us) client_negotiation.cc:240] Negotiation successful
0928 08:38:03.568940 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":1,"thread_start_us":788,"threads_started":1}
I20260928 08:38:03.569231 8505 raft_consensus.cc:504] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:03.569415 8290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc9d1d181414941a62d72706d4111df" candidate_uuid: "304d64805d4241d3914323de7627a8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef0262ddf5a43339816d2615819f557" is_pre_election: true
I20260928 08:38:03.569854 8290 raft_consensus.cc:2454] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 304d64805d4241d3914323de7627a8d7 in term 0.
I20260928 08:38:03.569974 8151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc9d1d181414941a62d72706d4111df" candidate_uuid: "304d64805d4241d3914323de7627a8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad3bbb2e240747db8627864f727715cf" is_pre_election: true
I20260928 08:38:03.570238 8151 raft_consensus.cc:2454] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 304d64805d4241d3914323de7627a8d7 in term 0.
I20260928 08:38:03.570514 8356 leader_election.cc:302] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 304d64805d4241d3914323de7627a8d7, 3ef0262ddf5a43339816d2615819f557; no voters:
I20260928 08:38:03.570911 8504 raft_consensus.cc:2781] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:38:03.571041 8504 raft_consensus.cc:482] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:38:03.571143 8504 raft_consensus.cc:3037] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:03.571165 8505 leader_election.cc:288] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ef0262ddf5a43339816d2615819f557 (127.0.97.2:44931), 304d64805d4241d3914323de7627a8d7 (127.0.97.3:43111)
I20260928 08:38:03.575192 8504 raft_consensus.cc:504] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:03.576097 8504 leader_election.cc:288] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [CANDIDATE]: Term 1 election: Requested vote from peers ad3bbb2e240747db8627864f727715cf (127.0.97.1:44399), 3ef0262ddf5a43339816d2615819f557 (127.0.97.2:44931)
I20260928 08:38:03.576656 8290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc9d1d181414941a62d72706d4111df" candidate_uuid: "304d64805d4241d3914323de7627a8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef0262ddf5a43339816d2615819f557"
I20260928 08:38:03.576889 8290 raft_consensus.cc:3037] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:03.580173 8151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc9d1d181414941a62d72706d4111df" candidate_uuid: "304d64805d4241d3914323de7627a8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad3bbb2e240747db8627864f727715cf"
I20260928 08:38:03.580442 8151 raft_consensus.cc:3037] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:03.580470 8290 raft_consensus.cc:2454] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 304d64805d4241d3914323de7627a8d7 in term 1.
I20260928 08:38:03.580899 8356 leader_election.cc:302] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 304d64805d4241d3914323de7627a8d7, 3ef0262ddf5a43339816d2615819f557; no voters:
I20260928 08:38:03.581208 8504 raft_consensus.cc:2781] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:38:03.583623 8151 raft_consensus.cc:2454] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 304d64805d4241d3914323de7627a8d7 in term 1.
I20260928 08:38:03.586721 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.572042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40657 (local address 127.0.97.2:44931)
0928 08:38:03.573044 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:03.573050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:03.576793 (+ 3743us) server_negotiation.cc:407] Connection header received
0928 08:38:03.577127 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:03.577136 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:03.577228 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:03.577302 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:38:03.578160 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.579366 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.584068 (+ 4702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.585430 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.585552 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.585684 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:03.586267 (+ 583us) server_negotiation.cc:299] Negotiation successful
0928 08:38:03.586438 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":891}
I20260928 08:38:03.586865 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.572225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48783 (local address 127.0.97.3:43111)
0928 08:38:03.572380 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:03.572385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:03.577019 (+ 4634us) server_negotiation.cc:407] Connection header received
0928 08:38:03.577240 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:03.577245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:03.577319 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:03.577377 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:38:03.578166 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.579343 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.584398 (+ 5055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.585665 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.585761 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.585865 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:03.586415 (+ 550us) server_negotiation.cc:299] Negotiation successful
0928 08:38:03.586586 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 08:38:03.587247 8504 raft_consensus.cc:686] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [term 1 LEADER]: Becoming Leader. State: Replica: 304d64805d4241d3914323de7627a8d7, State: Running, Role: LEADER
I20260928 08:38:03.587584 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.571700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43111 (local address 127.0.97.1:48783)
0928 08:38:03.576935 (+ 5235us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:03.576951 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:03.577099 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:03.577538 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:03.577541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:03.577557 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:38:03.578013 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.578021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.579521 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.579526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.584225 (+ 4699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.584240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.586050 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.586056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.586255 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.586284 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 08:38:03.587021 (+ 737us) client_negotiation.cc:240] Negotiation successful
0928 08:38:03.587191 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5119,"mutex_wait_us":2406,"thread_start_us":4671,"threads_started":1}
I20260928 08:38:03.588349 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.571626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44931 (local address 127.0.97.1:40657)
0928 08:38:03.576673 (+ 5047us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:03.576698 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:03.576899 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:03.577513 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:03.577519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:03.577544 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:38:03.577940 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.577956 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.579522 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.579526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.582992 (+ 3466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.583006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.585829 (+ 2823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.585835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.586034 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.586066 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 08:38:03.586150 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 08:38:03.587510 (+ 1360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4896,"mutex_wait_us":1335,"thread_start_us":7962,"threads_started":1}
I20260928 08:38:03.588245 8429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc9d1d181414941a62d72706d4111df" candidate_uuid: "ad3bbb2e240747db8627864f727715cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "304d64805d4241d3914323de7627a8d7" is_pre_election: true
I20260928 08:38:03.588125 8504 consensus_queue.cc:237] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:03.589154 8290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc9d1d181414941a62d72706d4111df" candidate_uuid: "ad3bbb2e240747db8627864f727715cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef0262ddf5a43339816d2615819f557" is_pre_election: true
I20260928 08:38:03.589475 8290 raft_consensus.cc:2379] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ad3bbb2e240747db8627864f727715cf in current term 1: Already voted for candidate 304d64805d4241d3914323de7627a8d7 in this term.
I20260928 08:38:03.590095 8077 leader_election.cc:302] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad3bbb2e240747db8627864f727715cf; no voters: 304d64805d4241d3914323de7627a8d7, 3ef0262ddf5a43339816d2615819f557
I20260928 08:38:03.590477 8505 raft_consensus.cc:2726] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 08:38:03.593151 7988 catalog_manager.cc:5495] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 304d64805d4241d3914323de7627a8d7 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "304d64805d4241d3914323de7627a8d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } health_report { overall_health: HEALTHY } } }
Sep 28 08:38:03 dist-test-slave-w6c0 krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584681, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:38:03.695883 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.680988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46243 (local address 127.0.0.1:53650)
0928 08:38:03.681847 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:03.681868 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:03.682006 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:03.682395 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:03.682398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:03.683068 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:03.683256 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.683265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.684741 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.684745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.685643 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.685652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.686655 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.686659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.686780 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.687468 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:03.687488 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:03.689526 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:38:03.693410 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:03.693420 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:03.693426 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:03.693861 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:03.694269 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:03.694272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:03.694274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:03.694404 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:03.695045 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:38:03.695051 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:38:03.695202 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 08:38:03.695334 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 08:38:03.695567 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":668,"threads_started":1}
I20260928 08:38:03.696420 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.680988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53650 (local address 127.0.97.62:46243)
0928 08:38:03.681488 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:03.681495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:03.681980 (+ 485us) server_negotiation.cc:407] Connection header received
0928 08:38:03.682122 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:03.682131 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:03.682217 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:03.682356 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:03.683416 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.684595 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.685783 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.686430 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.686607 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.689728 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:03.689735 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:03.689740 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:03.689776 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:03.693284 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:03.694000 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:03.694006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:03.694009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:03.694155 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:03.694541 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:03.694546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:03.694549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:03.694918 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:03.695072 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:03.695485 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 08:38:03.695901 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":259,"threads_started":1}
I20260928 08:38:03.706620 7988 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53650:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 08:38:03.707724 7988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:38:03.715045 8264 tablet_service.cc:1467] Processing CreateTablet for tablet 54905cd962504c5cbcbdfcf987605439 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e434509085154516876fb34e2173539f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:38:03.715180 8409 tablet_service.cc:1467] Processing CreateTablet for tablet 54905cd962504c5cbcbdfcf987605439 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e434509085154516876fb34e2173539f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:38:03.715659 8264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54905cd962504c5cbcbdfcf987605439. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:03.715759 8409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54905cd962504c5cbcbdfcf987605439. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:03.718199 8126 tablet_service.cc:1467] Processing CreateTablet for tablet 54905cd962504c5cbcbdfcf987605439 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e434509085154516876fb34e2173539f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:38:03.718782 8126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54905cd962504c5cbcbdfcf987605439. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:03.722929 8499 tablet_bootstrap.cc:492] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557: Bootstrap starting.
I20260928 08:38:03.723165 8500 tablet_bootstrap.cc:492] T 54905cd962504c5cbcbdfcf987605439 P 304d64805d4241d3914323de7627a8d7: Bootstrap starting.
I20260928 08:38:03.724728 8499 tablet_bootstrap.cc:654] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:03.724829 8500 tablet_bootstrap.cc:654] T 54905cd962504c5cbcbdfcf987605439 P 304d64805d4241d3914323de7627a8d7: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:03.726717 8498 tablet_bootstrap.cc:492] T 54905cd962504c5cbcbdfcf987605439 P ad3bbb2e240747db8627864f727715cf: Bootstrap starting.
I20260928 08:38:03.728788 8498 tablet_bootstrap.cc:654] T 54905cd962504c5cbcbdfcf987605439 P ad3bbb2e240747db8627864f727715cf: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:03.735483 8499 tablet_bootstrap.cc:492] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557: No bootstrap required, opened a new log
I20260928 08:38:03.735651 8500 tablet_bootstrap.cc:492] T 54905cd962504c5cbcbdfcf987605439 P 304d64805d4241d3914323de7627a8d7: No bootstrap required, opened a new log
I20260928 08:38:03.735704 8499 ts_tablet_manager.cc:1397] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260928 08:38:03.735833 8500 ts_tablet_manager.cc:1397] T 54905cd962504c5cbcbdfcf987605439 P 304d64805d4241d3914323de7627a8d7: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20260928 08:38:03.736618 8500 raft_consensus.cc:348] T 54905cd962504c5cbcbdfcf987605439 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:03.736707 8499 raft_consensus.cc:348] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:03.736877 8500 raft_consensus.cc:374] T 54905cd962504c5cbcbdfcf987605439 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:03.736953 8500 raft_consensus.cc:729] T 54905cd962504c5cbcbdfcf987605439 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 304d64805d4241d3914323de7627a8d7, State: Initialized, Role: FOLLOWER
I20260928 08:38:03.736974 8499 raft_consensus.cc:374] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:03.737056 8499 raft_consensus.cc:729] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef0262ddf5a43339816d2615819f557, State: Initialized, Role: FOLLOWER
I20260928 08:38:03.737226 8500 consensus_queue.cc:260] T 54905cd962504c5cbcbdfcf987605439 P 304d64805d4241d3914323de7627a8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:03.737363 8499 consensus_queue.cc:260] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:03.737660 8504 tablet_replica.cc:439] T 54905cd962504c5cbcbdfcf987605439 P 304d64805d4241d3914323de7627a8d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } } }
I20260928 08:38:03.737937 8504 tablet_replica.cc:442] T 54905cd962504c5cbcbdfcf987605439 P 304d64805d4241d3914323de7627a8d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:38:03.737794 8507 tablet_replica.cc:439] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } } }
I20260928 08:38:03.738126 8507 tablet_replica.cc:442] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:38:03.740231 8500 ts_tablet_manager.cc:1428] T 54905cd962504c5cbcbdfcf987605439 P 304d64805d4241d3914323de7627a8d7: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260928 08:38:03.740978 8498 tablet_bootstrap.cc:492] T 54905cd962504c5cbcbdfcf987605439 P ad3bbb2e240747db8627864f727715cf: No bootstrap required, opened a new log
I20260928 08:38:03.741153 8498 ts_tablet_manager.cc:1397] T 54905cd962504c5cbcbdfcf987605439 P ad3bbb2e240747db8627864f727715cf: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20260928 08:38:03.741840 8499 ts_tablet_manager.cc:1428] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260928 08:38:03.741922 8498 raft_consensus.cc:348] T 54905cd962504c5cbcbdfcf987605439 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:03.742149 8498 raft_consensus.cc:374] T 54905cd962504c5cbcbdfcf987605439 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:03.742234 8498 raft_consensus.cc:729] T 54905cd962504c5cbcbdfcf987605439 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad3bbb2e240747db8627864f727715cf, State: Initialized, Role: FOLLOWER
I20260928 08:38:03.742480 8498 consensus_queue.cc:260] T 54905cd962504c5cbcbdfcf987605439 P ad3bbb2e240747db8627864f727715cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:03.743788 8505 tablet_replica.cc:439] T 54905cd962504c5cbcbdfcf987605439 P ad3bbb2e240747db8627864f727715cf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } } }
I20260928 08:38:03.744060 8505 tablet_replica.cc:442] T 54905cd962504c5cbcbdfcf987605439 P ad3bbb2e240747db8627864f727715cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:38:03.744997 8498 ts_tablet_manager.cc:1428] T 54905cd962504c5cbcbdfcf987605439 P ad3bbb2e240747db8627864f727715cf: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 08:38:03.798226 8507 raft_consensus.cc:482] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:38:03.798512 8507 raft_consensus.cc:504] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:03.799271 8507 leader_election.cc:288] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad3bbb2e240747db8627864f727715cf (127.0.97.1:44399), 304d64805d4241d3914323de7627a8d7 (127.0.97.3:43111)
I20260928 08:38:03.799791 8429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54905cd962504c5cbcbdfcf987605439" candidate_uuid: "3ef0262ddf5a43339816d2615819f557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "304d64805d4241d3914323de7627a8d7" is_pre_election: true
I20260928 08:38:03.799829 8151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54905cd962504c5cbcbdfcf987605439" candidate_uuid: "3ef0262ddf5a43339816d2615819f557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad3bbb2e240747db8627864f727715cf" is_pre_election: true
I20260928 08:38:03.800079 8429 raft_consensus.cc:2454] T 54905cd962504c5cbcbdfcf987605439 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ef0262ddf5a43339816d2615819f557 in term 0.
I20260928 08:38:03.800079 8151 raft_consensus.cc:2454] T 54905cd962504c5cbcbdfcf987605439 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ef0262ddf5a43339816d2615819f557 in term 0.
I20260928 08:38:03.800513 8219 leader_election.cc:302] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ef0262ddf5a43339816d2615819f557, ad3bbb2e240747db8627864f727715cf; no voters:
I20260928 08:38:03.800884 8507 raft_consensus.cc:2781] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:38:03.800997 8507 raft_consensus.cc:482] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:38:03.801091 8507 raft_consensus.cc:3037] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:03.804163 8507 raft_consensus.cc:504] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:03.804792 8507 leader_election.cc:288] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 [CANDIDATE]: Term 1 election: Requested vote from peers ad3bbb2e240747db8627864f727715cf (127.0.97.1:44399), 304d64805d4241d3914323de7627a8d7 (127.0.97.3:43111)
I20260928 08:38:03.805222 8151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54905cd962504c5cbcbdfcf987605439" candidate_uuid: "3ef0262ddf5a43339816d2615819f557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad3bbb2e240747db8627864f727715cf"
I20260928 08:38:03.805269 8429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54905cd962504c5cbcbdfcf987605439" candidate_uuid: "3ef0262ddf5a43339816d2615819f557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "304d64805d4241d3914323de7627a8d7"
I20260928 08:38:03.805444 8151 raft_consensus.cc:3037] T 54905cd962504c5cbcbdfcf987605439 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:03.805495 8429 raft_consensus.cc:3037] T 54905cd962504c5cbcbdfcf987605439 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:03.808080 8429 raft_consensus.cc:2454] T 54905cd962504c5cbcbdfcf987605439 P 304d64805d4241d3914323de7627a8d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ef0262ddf5a43339816d2615819f557 in term 1.
I20260928 08:38:03.808107 8151 raft_consensus.cc:2454] T 54905cd962504c5cbcbdfcf987605439 P ad3bbb2e240747db8627864f727715cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ef0262ddf5a43339816d2615819f557 in term 1.
I20260928 08:38:03.808537 8216 leader_election.cc:302] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 304d64805d4241d3914323de7627a8d7, 3ef0262ddf5a43339816d2615819f557; no voters:
I20260928 08:38:03.808939 8507 raft_consensus.cc:2781] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:38:03.809139 8507 raft_consensus.cc:686] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 [term 1 LEADER]: Becoming Leader. State: Replica: 3ef0262ddf5a43339816d2615819f557, State: Running, Role: LEADER
I20260928 08:38:03.809397 8507 consensus_queue.cc:237] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:03.810907 8506 tablet_replica.cc:439] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557: TxnStatusTablet state changed. Reason: New leader 3ef0262ddf5a43339816d2615819f557. Latest consensus state: current_term: 1 leader_uuid: "3ef0262ddf5a43339816d2615819f557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } } }
I20260928 08:38:03.811185 8506 tablet_replica.cc:442] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:38:03.811817 8544 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 08:38:03.812063 8544 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 08:38:03.812506 7989 catalog_manager.cc:5495] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ef0262ddf5a43339816d2615819f557 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3ef0262ddf5a43339816d2615819f557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } health_report { overall_health: UNKNOWN } } }
I20260928 08:38:03.848398 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.842466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47478 (local address 127.0.97.2:44931)
0928 08:38:03.842706 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:03.842712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:03.842831 (+ 119us) server_negotiation.cc:407] Connection header received
0928 08:38:03.843078 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:03.843081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:03.843143 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:03.843226 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:38:03.843666 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.844416 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.846416 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.847026 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.847134 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.847727 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:38:03.847861 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:38:03.847930 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:03.848105 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 08:38:03.848171 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260928 08:38:03.848557 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.842434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44931 (local address 127.0.0.1:47478)
0928 08:38:03.842736 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:03.842758 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:03.842906 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:03.843273 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:03.843280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:03.843297 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:38:03.843495 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.843508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.844533 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.844538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.846276 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.846291 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.847208 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.847214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.847440 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.847522 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:38:03.847963 (+ 441us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:38:03.847983 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 08:38:03.848084 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 08:38:03.848238 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 08:38:03.852556 8151 raft_consensus.cc:1261] T 54905cd962504c5cbcbdfcf987605439 P ad3bbb2e240747db8627864f727715cf [term 1 FOLLOWER]: Refusing update from remote peer 3ef0262ddf5a43339816d2615819f557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:38:03.852823 8429 raft_consensus.cc:1261] T 54905cd962504c5cbcbdfcf987605439 P 304d64805d4241d3914323de7627a8d7 [term 1 FOLLOWER]: Refusing update from remote peer 3ef0262ddf5a43339816d2615819f557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:38:03.853569 8507 consensus_queue.cc:1035] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:38:03.853888 8506 consensus_queue.cc:1035] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:38:03.857056 8504 tablet_replica.cc:439] T 54905cd962504c5cbcbdfcf987605439 P 304d64805d4241d3914323de7627a8d7: TxnStatusTablet state changed. Reason: New leader 3ef0262ddf5a43339816d2615819f557. Latest consensus state: current_term: 1 leader_uuid: "3ef0262ddf5a43339816d2615819f557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } } }
I20260928 08:38:03.857389 8504 tablet_replica.cc:442] T 54905cd962504c5cbcbdfcf987605439 P 304d64805d4241d3914323de7627a8d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:38:03.859910 8546 mvcc.cc:205] Tried to move back new op lower bound from 7334234865054126080 to 7334234864885600256. Current Snapshot: MvccSnapshot[applied={T|T < 7334234865054126080}]
I20260928 08:38:03.860474 8506 tablet_replica.cc:439] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ef0262ddf5a43339816d2615819f557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } } }
I20260928 08:38:03.860762 8506 tablet_replica.cc:442] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:38:03.862725 8504 tablet_replica.cc:439] T 54905cd962504c5cbcbdfcf987605439 P 304d64805d4241d3914323de7627a8d7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ef0262ddf5a43339816d2615819f557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } } }
I20260928 08:38:03.863050 8504 tablet_replica.cc:442] T 54905cd962504c5cbcbdfcf987605439 P 304d64805d4241d3914323de7627a8d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:38:03.868981 8505 tablet_replica.cc:439] T 54905cd962504c5cbcbdfcf987605439 P ad3bbb2e240747db8627864f727715cf: TxnStatusTablet state changed. Reason: New leader 3ef0262ddf5a43339816d2615819f557. Latest consensus state: current_term: 1 leader_uuid: "3ef0262ddf5a43339816d2615819f557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } } }
I20260928 08:38:03.869264 8505 tablet_replica.cc:442] T 54905cd962504c5cbcbdfcf987605439 P ad3bbb2e240747db8627864f727715cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:38:03.870772 8506 tablet_replica.cc:439] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ef0262ddf5a43339816d2615819f557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } } }
I20260928 08:38:03.871057 8506 tablet_replica.cc:442] T 54905cd962504c5cbcbdfcf987605439 P 3ef0262ddf5a43339816d2615819f557: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:38:03.871893 8548 mvcc.cc:205] Tried to move back new op lower bound from 7334234865054126080 to 7334234864885600256. Current Snapshot: MvccSnapshot[applied={T|T < 7334234865054126080}]
I20260928 08:38:03.876705 8505 tablet_replica.cc:439] T 54905cd962504c5cbcbdfcf987605439 P ad3bbb2e240747db8627864f727715cf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ef0262ddf5a43339816d2615819f557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } } }
I20260928 08:38:03.876938 8505 tablet_replica.cc:442] T 54905cd962504c5cbcbdfcf987605439 P ad3bbb2e240747db8627864f727715cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:38:03.877794 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.868449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44176 (local address 127.0.97.3:43111)
0928 08:38:03.868621 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:03.868627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:03.870040 (+ 1413us) server_negotiation.cc:407] Connection header received
0928 08:38:03.870337 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:03.870342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:03.870440 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:03.870587 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:38:03.871254 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.872358 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.874837 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.875514 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.875609 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.876295 (+ 686us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:38:03.877064 (+ 769us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:38:03.877242 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:03.877452 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 08:38:03.877537 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:38:03.893543 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.888108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48301 (local address 127.0.97.2:44931)
0928 08:38:03.888263 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:03.888268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:03.888766 (+ 498us) server_negotiation.cc:407] Connection header received
0928 08:38:03.888985 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:03.888989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:03.889055 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:03.889147 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:38:03.889628 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.890393 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.891709 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.892150 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.892304 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.892734 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:38:03.892841 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:38:03.892941 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:03.893151 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 08:38:03.893235 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 08:38:03.893603 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.888125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44931 (local address 127.0.97.3:48301)
0928 08:38:03.888650 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:03.888666 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:03.888844 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:03.889173 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:03.889178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:03.889203 (+ 25us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:38:03.889485 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.889493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.890551 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.890557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.891561 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.891572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.892334 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.892340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.892483 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.892532 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:38:03.892954 (+ 422us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:38:03.892967 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 08:38:03.893114 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 08:38:03.893247 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":277,"threads_started":1}
I20260928 08:38:03.903671 8151 raft_consensus.cc:1261] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf [term 1 FOLLOWER]: Refusing update from remote peer 304d64805d4241d3914323de7627a8d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:38:03.903648 8290 raft_consensus.cc:1261] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557 [term 1 FOLLOWER]: Refusing update from remote peer 304d64805d4241d3914323de7627a8d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:38:03.904420 8531 consensus_queue.cc:1035] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:38:03.905007 8504 consensus_queue.cc:1035] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:38:03.937636 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.931262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49009 (local address 127.0.97.3:43111)
0928 08:38:03.931428 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:03.931436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:03.931923 (+ 487us) server_negotiation.cc:407] Connection header received
0928 08:38:03.932126 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:03.932131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:03.932233 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:03.932335 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:38:03.932841 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.933848 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.935847 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.936281 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.936429 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.936853 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:38:03.936971 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:38:03.937066 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:03.937232 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 08:38:03.937323 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:38:03.937757 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:03.931261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43111 (local address 127.0.97.2:49009)
0928 08:38:03.931824 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:03.931839 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:03.931990 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:03.932351 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:03.932355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:03.932366 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:38:03.932694 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.932703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.933997 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.934002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.935708 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:03.935721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:03.936458 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:03.936464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:03.936615 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:03.936677 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:38:03.937066 (+ 389us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:38:03.937096 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 08:38:03.937182 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 08:38:03.937327 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":307,"threads_started":1}
I20260928 08:38:03.977439 7989 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:38:03.977739 7989 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:38:03.981442 7989 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37: Sending DeleteTablet for 3 replicas of tablet bdc9d1d181414941a62d72706d4111df
I20260928 08:38:03.982404 8264 tablet_service.cc:1514] Processing DeleteTablet for tablet bdc9d1d181414941a62d72706d4111df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:38:03 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47460
I20260928 08:38:03.982463 8126 tablet_service.cc:1514] Processing DeleteTablet for tablet bdc9d1d181414941a62d72706d4111df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:38:03 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55786
I20260928 08:38:03.982697 8409 tablet_service.cc:1514] Processing DeleteTablet for tablet bdc9d1d181414941a62d72706d4111df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:38:03 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44166
I20260928 08:38:03.994387 8566 tablet_replica.cc:330] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557: stopping tablet replica
I20260928 08:38:03.994916 8568 tablet_replica.cc:330] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7: stopping tablet replica
I20260928 08:38:03.995245 8568 raft_consensus.cc:2229] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:38:03.995553 8566 raft_consensus.cc:2229] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:38:03.996279 8568 raft_consensus.cc:2258] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:38:03.996618 8566 raft_consensus.cc:2258] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:38:03.997922 8566 ts_tablet_manager.cc:1905] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:38:03.998991 8568 ts_tablet_manager.cc:1905] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:38:04.011418 8567 tablet_replica.cc:330] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf: stopping tablet replica
I20260928 08:38:04.011665 8567 raft_consensus.cc:2229] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:38:04.011960 8567 raft_consensus.cc:2258] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:38:04.014312 8566 ts_tablet_manager.cc:1918] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:38:04.014532 8568 ts_tablet_manager.cc:1918] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:38:04.014730 8568 log.cc:1196] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/bdc9d1d181414941a62d72706d4111df
I20260928 08:38:04.015030 8566 log.cc:1196] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/bdc9d1d181414941a62d72706d4111df
I20260928 08:38:04.015728 8567 ts_tablet_manager.cc:1905] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:38:04.015986 8566 ts_tablet_manager.cc:1939] T bdc9d1d181414941a62d72706d4111df P 3ef0262ddf5a43339816d2615819f557: Deleting consensus metadata
I20260928 08:38:04.017248 7974 catalog_manager.cc:4841] TS 3ef0262ddf5a43339816d2615819f557 (127.0.97.2:44931): tablet bdc9d1d181414941a62d72706d4111df (table test-table [id=0d6ef754cd8e4b53a906c13ea358e730]) successfully deleted
I20260928 08:38:04.017762 8568 ts_tablet_manager.cc:1939] T bdc9d1d181414941a62d72706d4111df P 304d64805d4241d3914323de7627a8d7: Deleting consensus metadata
I20260928 08:38:04.018975 7973 catalog_manager.cc:4841] TS 304d64805d4241d3914323de7627a8d7 (127.0.97.3:43111): tablet bdc9d1d181414941a62d72706d4111df (table test-table [id=0d6ef754cd8e4b53a906c13ea358e730]) successfully deleted
I20260928 08:38:04.022948 8567 ts_tablet_manager.cc:1918] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:38:04.023066 8567 log.cc:1196] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/wal/wals/bdc9d1d181414941a62d72706d4111df
I20260928 08:38:04.023481 8567 ts_tablet_manager.cc:1939] T bdc9d1d181414941a62d72706d4111df P ad3bbb2e240747db8627864f727715cf: Deleting consensus metadata
I20260928 08:38:04.024438 7976 catalog_manager.cc:4841] TS ad3bbb2e240747db8627864f727715cf (127.0.97.1:44399): tablet bdc9d1d181414941a62d72706d4111df (table test-table [id=0d6ef754cd8e4b53a906c13ea358e730]) successfully deleted
Sep 28 08:38:04 dist-test-slave-w6c0 krb5kdc[7944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:38:04 dist-test-slave-w6c0 krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:04 dist-test-slave-w6c0 krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:38:04.115868 8538 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:04.095530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.0.97.62:46243)
0928 08:38:04.095768 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:04.095775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:04.102647 (+ 6872us) server_negotiation.cc:407] Connection header received
0928 08:38:04.103110 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:04.103116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:04.103216 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:04.103299 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:04.105125 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:04.106204 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:04.108053 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:04.109153 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:04.109267 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:04.115611 (+ 6344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53662: BlockingRecv error: recv got EOF from 127.0.0.1:53662 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 08:38:04 dist-test-slave-w6c0 krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:38:04.140002 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:04.124948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53668 (local address 127.0.97.62:46243)
0928 08:38:04.125181 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:04.125188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:04.125645 (+ 457us) server_negotiation.cc:407] Connection header received
0928 08:38:04.125856 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:04.125861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:04.125939 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:04.126057 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:04.127488 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:04.128482 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:04.129735 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:04.130152 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:04.130298 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:04.134553 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:04.134560 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:04.134565 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:04.134607 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:04.137303 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:04.137921 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:04.137929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:04.137938 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:04.138063 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:04.138491 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:04.138498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:04.138574 (+ 76us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:04.138835 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:04.139109 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:04.139405 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 08:38:04.139640 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 08:38:04.143597 7989 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:38:04.144266 7989 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:38:04.151130 8126 tablet_service.cc:1467] Processing CreateTablet for tablet edccca8affbd437193f762646586b364 (DEFAULT_TABLE table=test-table [id=f341dad663c44f1892e35713ca2318eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:38:04.151594 8126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edccca8affbd437193f762646586b364. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:04.151561 8409 tablet_service.cc:1467] Processing CreateTablet for tablet edccca8affbd437193f762646586b364 (DEFAULT_TABLE table=test-table [id=f341dad663c44f1892e35713ca2318eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:38:04.151975 8409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edccca8affbd437193f762646586b364. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:04.155332 8264 tablet_service.cc:1467] Processing CreateTablet for tablet edccca8affbd437193f762646586b364 (DEFAULT_TABLE table=test-table [id=f341dad663c44f1892e35713ca2318eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:38:04.155762 8264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edccca8affbd437193f762646586b364. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:04.157953 8498 tablet_bootstrap.cc:492] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf: Bootstrap starting.
I20260928 08:38:04.159569 8498 tablet_bootstrap.cc:654] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:04.169370 8500 tablet_bootstrap.cc:492] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7: Bootstrap starting.
I20260928 08:38:04.171226 8500 tablet_bootstrap.cc:654] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:04.171983 8499 tablet_bootstrap.cc:492] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557: Bootstrap starting.
I20260928 08:38:04.174082 8499 tablet_bootstrap.cc:654] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:04.187404 8499 tablet_bootstrap.cc:492] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557: No bootstrap required, opened a new log
I20260928 08:38:04.188959 8499 ts_tablet_manager.cc:1397] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20260928 08:38:04.189994 8500 tablet_bootstrap.cc:492] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7: No bootstrap required, opened a new log
I20260928 08:38:04.190164 8500 ts_tablet_manager.cc:1397] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.006s
I20260928 08:38:04.190378 8499 raft_consensus.cc:348] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:04.190867 8499 raft_consensus.cc:374] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:04.191032 8499 raft_consensus.cc:729] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef0262ddf5a43339816d2615819f557, State: Initialized, Role: FOLLOWER
I20260928 08:38:04.190961 8500 raft_consensus.cc:348] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:04.191258 8500 raft_consensus.cc:374] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:04.191365 8500 raft_consensus.cc:729] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 304d64805d4241d3914323de7627a8d7, State: Initialized, Role: FOLLOWER
I20260928 08:38:04.191627 8500 consensus_queue.cc:260] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:04.193969 8499 consensus_queue.cc:260] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:04.194705 8500 ts_tablet_manager.cc:1428] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 08:38:04.199154 8499 ts_tablet_manager.cc:1428] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20260928 08:38:04.203634 8498 tablet_bootstrap.cc:492] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf: No bootstrap required, opened a new log
I20260928 08:38:04.203832 8498 ts_tablet_manager.cc:1397] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf: Time spent bootstrapping tablet: real 0.046s user 0.022s sys 0.021s
I20260928 08:38:04.204661 8498 raft_consensus.cc:348] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:04.204936 8498 raft_consensus.cc:374] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:04.205039 8498 raft_consensus.cc:729] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad3bbb2e240747db8627864f727715cf, State: Initialized, Role: FOLLOWER
I20260928 08:38:04.205291 8498 consensus_queue.cc:260] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:04.205914 8498 ts_tablet_manager.cc:1428] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 08:38:04.221814 8505 raft_consensus.cc:482] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:38:04.222016 8505 raft_consensus.cc:504] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:04.222785 8505 leader_election.cc:288] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ef0262ddf5a43339816d2615819f557 (127.0.97.2:44931), 304d64805d4241d3914323de7627a8d7 (127.0.97.3:43111)
I20260928 08:38:04.223284 8290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edccca8affbd437193f762646586b364" candidate_uuid: "ad3bbb2e240747db8627864f727715cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef0262ddf5a43339816d2615819f557" is_pre_election: true
I20260928 08:38:04.223538 8290 raft_consensus.cc:2454] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad3bbb2e240747db8627864f727715cf in term 0.
I20260928 08:38:04.223687 8429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edccca8affbd437193f762646586b364" candidate_uuid: "ad3bbb2e240747db8627864f727715cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "304d64805d4241d3914323de7627a8d7" is_pre_election: true
I20260928 08:38:04.223915 8429 raft_consensus.cc:2454] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad3bbb2e240747db8627864f727715cf in term 0.
I20260928 08:38:04.224406 8076 leader_election.cc:302] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 304d64805d4241d3914323de7627a8d7, ad3bbb2e240747db8627864f727715cf; no voters:
I20260928 08:38:04.224828 8505 raft_consensus.cc:2781] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:38:04.224963 8505 raft_consensus.cc:482] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:38:04.225062 8505 raft_consensus.cc:3037] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:04.227396 8505 raft_consensus.cc:504] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:04.228374 8505 leader_election.cc:288] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [CANDIDATE]: Term 1 election: Requested vote from peers 3ef0262ddf5a43339816d2615819f557 (127.0.97.2:44931), 304d64805d4241d3914323de7627a8d7 (127.0.97.3:43111)
I20260928 08:38:04.228511 8429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edccca8affbd437193f762646586b364" candidate_uuid: "ad3bbb2e240747db8627864f727715cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "304d64805d4241d3914323de7627a8d7"
I20260928 08:38:04.228760 8429 raft_consensus.cc:3037] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:04.230778 8290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edccca8affbd437193f762646586b364" candidate_uuid: "ad3bbb2e240747db8627864f727715cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef0262ddf5a43339816d2615819f557"
I20260928 08:38:04.230993 8290 raft_consensus.cc:3037] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:04.232191 8429 raft_consensus.cc:2454] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad3bbb2e240747db8627864f727715cf in term 1.
I20260928 08:38:04.232692 8076 leader_election.cc:302] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 304d64805d4241d3914323de7627a8d7, ad3bbb2e240747db8627864f727715cf; no voters:
I20260928 08:38:04.233131 8505 raft_consensus.cc:2781] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:38:04.234793 8290 raft_consensus.cc:2454] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad3bbb2e240747db8627864f727715cf in term 1.
I20260928 08:38:04.235754 8505 raft_consensus.cc:686] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [term 1 LEADER]: Becoming Leader. State: Replica: ad3bbb2e240747db8627864f727715cf, State: Running, Role: LEADER
I20260928 08:38:04.236249 8505 consensus_queue.cc:237] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:04.239964 7989 catalog_manager.cc:5495] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ad3bbb2e240747db8627864f727715cf (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ad3bbb2e240747db8627864f727715cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } health_report { overall_health: UNKNOWN } } }
I20260928 08:38:04.256157 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:04.248945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55794 (local address 127.0.97.1:44399)
0928 08:38:04.249404 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:04.249411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:04.249621 (+ 210us) server_negotiation.cc:407] Connection header received
0928 08:38:04.249865 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:04.249871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:04.249979 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:04.250172 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:38:04.250561 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:04.251401 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:04.253664 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:04.254205 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:04.254439 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:04.254926 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:38:04.255499 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:38:04.255657 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:04.255736 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 08:38:04.255816 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":250,"threads_started":1}
I20260928 08:38:04.259929 8429 raft_consensus.cc:1261] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7 [term 1 FOLLOWER]: Refusing update from remote peer ad3bbb2e240747db8627864f727715cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:38:04.259910 8290 raft_consensus.cc:1261] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557 [term 1 FOLLOWER]: Refusing update from remote peer ad3bbb2e240747db8627864f727715cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:38:04.260640 8584 consensus_queue.cc:1035] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:38:04.260962 8505 consensus_queue.cc:1035] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:38:04.297001 7988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:38:04.297290 7988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:38:04.301663 7988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37: Sending DeleteTablet for 3 replicas of tablet edccca8affbd437193f762646586b364
I20260928 08:38:04.302821 8409 tablet_service.cc:1514] Processing DeleteTablet for tablet edccca8affbd437193f762646586b364 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:38:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44166
I20260928 08:38:04.303158 8568 tablet_replica.cc:330] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7: stopping tablet replica
I20260928 08:38:04.303393 8568 raft_consensus.cc:2229] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:38:04.303579 8568 pending_rounds.cc:62] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7: Trying to abort 2 pending ops.
I20260928 08:38:04.303864 8126 tablet_service.cc:1514] Processing DeleteTablet for tablet edccca8affbd437193f762646586b364 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:38:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55786
I20260928 08:38:04.303692 8568 pending_rounds.cc:68] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7334234866720395264 op_type: WRITE_OP write_request { tablet_id: "edccca8affbd437193f762646586b364" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\230\321\350\325\006\"8\n\ntest-admin\022*\n f341dad663c44f1892e35713ca2318eb\020\001\030\001 \001(\001" signature: "<redacted>""y\303eA\330\310\223\265y0\240\3415\352L/\037\363J\350\"\343rlKc[\277\347;\343\316\200\273\"}\353H\022\206-\'\2467eB@\207\201\315\005^\023A\260\230\001g\327\351tG\005\005" signing_key_seq_num: 0 } } request_id { client_id: "cdeef03408b946d38e09c38202896f1d" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260928 08:38:04.304176 8567 tablet_replica.cc:330] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf: stopping tablet replica
I20260928 08:38:04.304364 8567 raft_consensus.cc:2229] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:38:04.304361 8568 pending_rounds.cc:68] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334234866636148736 op_type: NO_OP noop_request { }
I20260928 08:38:04.304513 8568 raft_consensus.cc:2866] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260928 08:38:04.304663 8568 raft_consensus.cc:2258] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:38:04.305536 8264 tablet_service.cc:1514] Processing DeleteTablet for tablet edccca8affbd437193f762646586b364 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:38:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47460
I20260928 08:38:04.305838 8566 tablet_replica.cc:330] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557: stopping tablet replica
I20260928 08:38:04.306061 8566 raft_consensus.cc:2229] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:38:04.306324 8566 raft_consensus.cc:2258] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:38:04.307304 8568 ts_tablet_manager.cc:1905] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:38:04.308122 8566 ts_tablet_manager.cc:1905] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:38:04.311899 7988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:38:04.312731 7988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:38:04.317713 8566 ts_tablet_manager.cc:1918] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:38:04.317885 8566 log.cc:1196] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/edccca8affbd437193f762646586b364
I20260928 08:38:04.318360 8566 ts_tablet_manager.cc:1939] T edccca8affbd437193f762646586b364 P 3ef0262ddf5a43339816d2615819f557: Deleting consensus metadata
I20260928 08:38:04.321007 8126 tablet_service.cc:1467] Processing CreateTablet for tablet fa14e3a84c3344b49331ddc9e66361c4 (DEFAULT_TABLE table=test-table [id=151084bb6aaf4535b94c74e68591f2cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:38:04.321115 8409 tablet_service.cc:1467] Processing CreateTablet for tablet fa14e3a84c3344b49331ddc9e66361c4 (DEFAULT_TABLE table=test-table [id=151084bb6aaf4535b94c74e68591f2cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:38:04.321468 8126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa14e3a84c3344b49331ddc9e66361c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:04.321516 8409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa14e3a84c3344b49331ddc9e66361c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:04.325338 8264 tablet_service.cc:1467] Processing CreateTablet for tablet fa14e3a84c3344b49331ddc9e66361c4 (DEFAULT_TABLE table=test-table [id=151084bb6aaf4535b94c74e68591f2cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:38:04.325764 8264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa14e3a84c3344b49331ddc9e66361c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:04.327945 8498 tablet_bootstrap.cc:492] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf: Bootstrap starting.
I20260928 08:38:04.329559 8498 tablet_bootstrap.cc:654] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:04.331071 8567 raft_consensus.cc:2258] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:38:04.332413 8567 ts_tablet_manager.cc:1905] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:38:04.335687 8498 tablet_bootstrap.cc:492] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf: No bootstrap required, opened a new log
I20260928 08:38:04.335844 8498 ts_tablet_manager.cc:1397] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260928 08:38:04.336668 8498 raft_consensus.cc:348] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:04.336930 8498 raft_consensus.cc:374] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:04.337018 8498 raft_consensus.cc:729] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad3bbb2e240747db8627864f727715cf, State: Initialized, Role: FOLLOWER
I20260928 08:38:04.337312 8498 consensus_queue.cc:260] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:04.337826 8498 ts_tablet_manager.cc:1428] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 08:38:04.338682 8591 raft_consensus.cc:482] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:38:04.338866 8591 raft_consensus.cc:504] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:04.339358 7974 catalog_manager.cc:4841] TS 3ef0262ddf5a43339816d2615819f557 (127.0.97.2:44931): tablet edccca8affbd437193f762646586b364 (table test-table [id=f341dad663c44f1892e35713ca2318eb]) successfully deleted
I20260928 08:38:04.339619 8591 leader_election.cc:288] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ef0262ddf5a43339816d2615819f557 (127.0.97.2:44931), 304d64805d4241d3914323de7627a8d7 (127.0.97.3:43111)
W20260928 08:38:04.341013 8076 leader_election.cc:341] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 304d64805d4241d3914323de7627a8d7 (127.0.97.3:43111): Not found: Tablet not found: fa14e3a84c3344b49331ddc9e66361c4
W20260928 08:38:04.341243 8077 leader_election.cc:341] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3ef0262ddf5a43339816d2615819f557 (127.0.97.2:44931): Not found: Tablet not found: fa14e3a84c3344b49331ddc9e66361c4
I20260928 08:38:04.341372 8077 leader_election.cc:302] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad3bbb2e240747db8627864f727715cf; no voters: 304d64805d4241d3914323de7627a8d7, 3ef0262ddf5a43339816d2615819f557
I20260928 08:38:04.341619 8591 raft_consensus.cc:2726] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 08:38:04.345463 8499 tablet_bootstrap.cc:492] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557: Bootstrap starting.
I20260928 08:38:04.347366 8499 tablet_bootstrap.cc:654] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:04.348765 8567 ts_tablet_manager.cc:1918] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:38:04.349350 8567 log.cc:1196] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/wal/wals/edccca8affbd437193f762646586b364
I20260928 08:38:04.350371 8567 ts_tablet_manager.cc:1939] T edccca8affbd437193f762646586b364 P ad3bbb2e240747db8627864f727715cf: Deleting consensus metadata
I20260928 08:38:04.351722 8568 ts_tablet_manager.cc:1918] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:38:04.352893 7976 catalog_manager.cc:4841] TS ad3bbb2e240747db8627864f727715cf (127.0.97.1:44399): tablet edccca8affbd437193f762646586b364 (table test-table [id=f341dad663c44f1892e35713ca2318eb]) successfully deleted
I20260928 08:38:04.353269 8568 log.cc:1196] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/edccca8affbd437193f762646586b364
I20260928 08:38:04.353854 8568 ts_tablet_manager.cc:1939] T edccca8affbd437193f762646586b364 P 304d64805d4241d3914323de7627a8d7: Deleting consensus metadata
I20260928 08:38:04.353986 8500 tablet_bootstrap.cc:492] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7: Bootstrap starting.
I20260928 08:38:04.355746 8500 tablet_bootstrap.cc:654] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:04.358580 7973 catalog_manager.cc:4841] TS 304d64805d4241d3914323de7627a8d7 (127.0.97.3:43111): tablet edccca8affbd437193f762646586b364 (table test-table [id=f341dad663c44f1892e35713ca2318eb]) successfully deleted
I20260928 08:38:04.360190 8499 tablet_bootstrap.cc:492] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557: No bootstrap required, opened a new log
I20260928 08:38:04.360378 8499 ts_tablet_manager.cc:1397] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20260928 08:38:04.361192 8499 raft_consensus.cc:348] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:04.361450 8499 raft_consensus.cc:374] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:04.361544 8499 raft_consensus.cc:729] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef0262ddf5a43339816d2615819f557, State: Initialized, Role: FOLLOWER
I20260928 08:38:04.361824 8499 consensus_queue.cc:260] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:04.365901 8499 ts_tablet_manager.cc:1428] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260928 08:38:04.368221 8500 tablet_bootstrap.cc:492] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7: No bootstrap required, opened a new log
I20260928 08:38:04.368422 8500 ts_tablet_manager.cc:1397] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20260928 08:38:04.369169 8500 raft_consensus.cc:348] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:04.369436 8500 raft_consensus.cc:374] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:04.369531 8500 raft_consensus.cc:729] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 304d64805d4241d3914323de7627a8d7, State: Initialized, Role: FOLLOWER
I20260928 08:38:04.369801 8500 consensus_queue.cc:260] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:04.370329 8500 ts_tablet_manager.cc:1428] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 08:38:04.390906 8504 raft_consensus.cc:482] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:38:04.391129 8504 raft_consensus.cc:504] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:04.391893 8504 leader_election.cc:288] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad3bbb2e240747db8627864f727715cf (127.0.97.1:44399), 3ef0262ddf5a43339816d2615819f557 (127.0.97.2:44931)
I20260928 08:38:04.392658 8289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa14e3a84c3344b49331ddc9e66361c4" candidate_uuid: "304d64805d4241d3914323de7627a8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef0262ddf5a43339816d2615819f557" is_pre_election: true
I20260928 08:38:04.392956 8289 raft_consensus.cc:2454] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 304d64805d4241d3914323de7627a8d7 in term 0.
I20260928 08:38:04.393008 8151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa14e3a84c3344b49331ddc9e66361c4" candidate_uuid: "304d64805d4241d3914323de7627a8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad3bbb2e240747db8627864f727715cf" is_pre_election: true
I20260928 08:38:04.393244 8151 raft_consensus.cc:2454] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 304d64805d4241d3914323de7627a8d7 in term 0.
I20260928 08:38:04.393810 8356 leader_election.cc:302] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 304d64805d4241d3914323de7627a8d7, 3ef0262ddf5a43339816d2615819f557; no voters:
I20260928 08:38:04.394655 8504 raft_consensus.cc:2781] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:38:04.394783 8504 raft_consensus.cc:482] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:38:04.394897 8504 raft_consensus.cc:3037] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:04.398059 8504 raft_consensus.cc:504] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:04.400437 8151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa14e3a84c3344b49331ddc9e66361c4" candidate_uuid: "304d64805d4241d3914323de7627a8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad3bbb2e240747db8627864f727715cf"
I20260928 08:38:04.400687 8151 raft_consensus.cc:3037] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:04.403925 8151 raft_consensus.cc:2454] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 304d64805d4241d3914323de7627a8d7 in term 1.
I20260928 08:38:04.404412 8358 leader_election.cc:302] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 304d64805d4241d3914323de7627a8d7, ad3bbb2e240747db8627864f727715cf; no voters:
I20260928 08:38:04.404734 8531 raft_consensus.cc:2781] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:38:04.407958 8289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa14e3a84c3344b49331ddc9e66361c4" candidate_uuid: "304d64805d4241d3914323de7627a8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef0262ddf5a43339816d2615819f557"
I20260928 08:38:04.408209 8289 raft_consensus.cc:3037] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:04.411257 8289 raft_consensus.cc:2454] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 304d64805d4241d3914323de7627a8d7 in term 1.
I20260928 08:38:04.413875 8504 leader_election.cc:288] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [CANDIDATE]: Term 1 election: Requested vote from peers ad3bbb2e240747db8627864f727715cf (127.0.97.1:44399), 3ef0262ddf5a43339816d2615819f557 (127.0.97.2:44931)
I20260928 08:38:04.422619 8531 raft_consensus.cc:686] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [term 1 LEADER]: Becoming Leader. State: Replica: 304d64805d4241d3914323de7627a8d7, State: Running, Role: LEADER
I20260928 08:38:04.422984 8531 consensus_queue.cc:237] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } }
I20260928 08:38:04.426918 7989 catalog_manager.cc:5495] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 304d64805d4241d3914323de7627a8d7 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "304d64805d4241d3914323de7627a8d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "304d64805d4241d3914323de7627a8d7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43111 } health_report { overall_health: HEALTHY } } }
I20260928 08:38:04.467217 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:04.461352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52852 (local address 127.0.97.3:43111)
0928 08:38:04.462040 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:04.462046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:04.462065 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:38:04.462128 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:04.462131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:04.462196 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:04.462310 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:38:04.462760 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:04.463739 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:04.465279 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:04.465744 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:04.465904 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:04.466365 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:38:04.466487 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:38:04.466644 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:04.466763 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 08:38:04.466846 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":392,"threads_started":1}
I20260928 08:38:04.481663 8289 raft_consensus.cc:1261] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557 [term 1 FOLLOWER]: Refusing update from remote peer 304d64805d4241d3914323de7627a8d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:38:04.482390 8151 raft_consensus.cc:1261] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf [term 1 FOLLOWER]: Refusing update from remote peer 304d64805d4241d3914323de7627a8d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:38:04.482259 8595 consensus_queue.cc:1035] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ef0262ddf5a43339816d2615819f557" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:38:04.482992 8595 consensus_queue.cc:1035] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad3bbb2e240747db8627864f727715cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:38:04.614454 7988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:38:04.614804 7988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:38:04.618189 7988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 183de3d8e93544c4a2c86d01a29dae37: Sending DeleteTablet for 3 replicas of tablet fa14e3a84c3344b49331ddc9e66361c4
I20260928 08:38:04.619079 8126 tablet_service.cc:1514] Processing DeleteTablet for tablet fa14e3a84c3344b49331ddc9e66361c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:38:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55786
I20260928 08:38:04.619418 8567 tablet_replica.cc:330] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf: stopping tablet replica
I20260928 08:38:04.619620 8567 raft_consensus.cc:2229] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:38:04.619675 8264 tablet_service.cc:1514] Processing DeleteTablet for tablet fa14e3a84c3344b49331ddc9e66361c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:38:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47460
I20260928 08:38:04.619920 8567 raft_consensus.cc:2258] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:38:04.619971 8566 tablet_replica.cc:330] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557: stopping tablet replica
I20260928 08:38:04.620014 8409 tablet_service.cc:1514] Processing DeleteTablet for tablet fa14e3a84c3344b49331ddc9e66361c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:38:04 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44166
I20260928 08:38:04.620154 8566 raft_consensus.cc:2229] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:38:04.620286 8568 tablet_replica.cc:330] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7: stopping tablet replica
I20260928 08:38:04.620402 8566 raft_consensus.cc:2258] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:38:04.620462 8568 raft_consensus.cc:2229] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:38:04.620903 8568 raft_consensus.cc:2258] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:38:04.621552 8566 ts_tablet_manager.cc:1905] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:38:04.621999 8568 ts_tablet_manager.cc:1905] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:38:04.622165 8567 ts_tablet_manager.cc:1905] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:38:04.624202 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 8060
I20260928 08:38:04.628041 8566 ts_tablet_manager.cc:1918] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:38:04.628186 8566 log.cc:1196] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/fa14e3a84c3344b49331ddc9e66361c4
I20260928 08:38:04.628602 8566 ts_tablet_manager.cc:1939] T fa14e3a84c3344b49331ddc9e66361c4 P 3ef0262ddf5a43339816d2615819f557: Deleting consensus metadata
I20260928 08:38:04.629696 7974 catalog_manager.cc:4841] TS 3ef0262ddf5a43339816d2615819f557 (127.0.97.2:44931): tablet fa14e3a84c3344b49331ddc9e66361c4 (table test-table [id=151084bb6aaf4535b94c74e68591f2cb]) successfully deleted
I20260928 08:38:04.629858 8568 ts_tablet_manager.cc:1918] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:38:04.630065 8568 log.cc:1196] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/fa14e3a84c3344b49331ddc9e66361c4
I20260928 08:38:04.630656 8568 ts_tablet_manager.cc:1939] T fa14e3a84c3344b49331ddc9e66361c4 P 304d64805d4241d3914323de7627a8d7: Deleting consensus metadata
I20260928 08:38:04.631752 7973 catalog_manager.cc:4841] TS 304d64805d4241d3914323de7627a8d7 (127.0.97.3:43111): tablet fa14e3a84c3344b49331ddc9e66361c4 (table test-table [id=151084bb6aaf4535b94c74e68591f2cb]) successfully deleted
I20260928 08:38:04.633292 8567 ts_tablet_manager.cc:1918] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:38:04.633453 8567 log.cc:1196] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584618551208-388-0/minicluster-data/ts-0/wal/wals/fa14e3a84c3344b49331ddc9e66361c4
I20260928 08:38:04.633904 8567 ts_tablet_manager.cc:1939] T fa14e3a84c3344b49331ddc9e66361c4 P ad3bbb2e240747db8627864f727715cf: Deleting consensus metadata
I20260928 08:38:04.638989 7976 catalog_manager.cc:4841] TS ad3bbb2e240747db8627864f727715cf (127.0.97.1:44399): tablet fa14e3a84c3344b49331ddc9e66361c4 (table test-table [id=151084bb6aaf4535b94c74e68591f2cb]) successfully deleted
Sep 28 08:38:04 dist-test-slave-w6c0 krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260928 08:38:04.654675 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:04.625131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50026 (local address 127.0.97.1:44399)
0928 08:38:04.625304 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:04.625311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:04.639470 (+ 14159us) server_negotiation.cc:407] Connection header received
0928 08:38:04.639699 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:04.639704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:04.639777 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:04.639894 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:04.641121 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:04.641948 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:04.643266 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:04.643812 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:04.644003 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:04.648287 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:04.648298 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:04.648317 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:04.648355 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:04.651505 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:04.652321 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:04.652329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:04.652331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:04.652468 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:04.652924 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:04.652929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:04.652931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:04.653316 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:04.653648 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:04.653966 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 08:38:04.654196 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:38:04.655421 8191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:04.778355 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 8060
I20260928 08:38:04.799682 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 8200
Sep 28 08:38:04 dist-test-slave-w6c0 krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260928 08:38:04.815090 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:04.800533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44260 (local address 127.0.97.2:44931)
0928 08:38:04.801137 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:04.801143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:04.801160 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:38:04.801213 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:04.801217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:04.801278 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:04.801370 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:04.802737 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:04.803467 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:04.804603 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:04.804983 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:04.805082 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:04.809241 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:04.809252 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:04.809268 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:04.809299 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:04.812233 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:04.813054 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:04.813061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:04.813064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:04.813220 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:04.813639 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:04.813643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:04.813645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:04.813971 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:04.814193 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:04.814470 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 08:38:04.814706 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":353,"threads_started":1}
I20260928 08:38:04.815778 8330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:04.965334 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 8200
I20260928 08:38:04.991528 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 8339
Sep 28 08:38:04 dist-test-slave-w6c0 krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260928 08:38:05.008759 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:04.992462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.0.97.3:43111)
0928 08:38:04.992946 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:04.992950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:04.992966 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:38:04.993020 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:04.993024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:04.993086 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:04.993180 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:04.994478 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:04.995323 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:04.996613 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:04.997247 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:04.997346 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:05.002730 (+ 5384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:05.002747 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:05.002764 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:05.002802 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:05.005755 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:05.006473 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:05.006488 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:05.006491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:05.006766 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:05.007143 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:05.007151 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:05.007156 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:05.007526 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:05.007853 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:05.008159 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 08:38:05.008398 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":241,"threads_started":1}
I20260928 08:38:05.009495 8469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:05.146528 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 8339
I20260928 08:38:05.168821 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 7957
I20260928 08:38:05.170337 8029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:05.291749 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 7957
2026-09-28T08:38:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5646 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:38:05 dist-test-slave-w6c0 krb5kdc[8615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:38:05 dist-test-slave-w6c0 krb5kdc[8615](info): set up 2 sockets
Sep 28 08:38:05 dist-test-slave-w6c0 krb5kdc[8615](info): commencing operation
krb5kdc: starting...
W20260928 08:38:06.431725 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.009s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:38:06 dist-test-slave-w6c0 krb5kdc[8615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:38:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:38:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:38:06.666006 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41829
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:38:06.993081 8628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:06.993425 8628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:06.993584 8628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:07.001062 8628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:38:07.001209 8628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:07.001288 8628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:07.001397 8628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:38:07.001489 8628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:38:07.011626 8628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:07.012981 8628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:07.014281 8628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:07.026454 8634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:07.026430 8633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:07.026633 8636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:07.027351 8628 server_base.cc:1034] running on GCE node
I20260928 08:38:07.028193 8628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:07.029626 8628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:07.030829 8628 hybrid_clock.cc:648] HybridClock initialized: now 1790584687030795 us; error 29 us; skew 500 ppm
I20260928 08:38:07.031286 8628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T08:38:07Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1796 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:38:07 dist-test-slave-w6c0 krb5kdc[8642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:38:07 dist-test-slave-w6c0 krb5kdc[8642](info): set up 2 sockets
Sep 28 08:38:07 dist-test-slave-w6c0 krb5kdc[8642](info): commencing operation
krb5kdc: starting...
W20260928 08:38:08.203930 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:38:08 dist-test-slave-w6c0 krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:38:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:38:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:38:08.467177 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42985
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:38:08.797693 8655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:08.798050 8655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:08.798202 8655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:08.806401 8655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:38:08.806648 8655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:08.806756 8655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:08.806859 8655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:38:08.806939 8655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:38:08.819840 8655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:08.821411 8655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:08.822878 8655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:08.834967 8660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:08.835024 8661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:08.835024 8663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:08.836019 8655 server_base.cc:1034] running on GCE node
I20260928 08:38:08.837244 8655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:08.839255 8655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:08.840567 8655 hybrid_clock.cc:648] HybridClock initialized: now 1790584688840443 us; error 78 us; skew 500 ppm
I20260928 08:38:08.841136 8655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:38:08 dist-test-slave-w6c0 krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:38:08.859620 8655 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:38:08.861348 8655 webserver.cc:460] Webserver started at http://127.0.97.62:42615/ using document root <none> and password file <none>
I20260928 08:38:08.862051 8655 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:08.862176 8655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:08.862457 8655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:08.865574 8655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "90a1d5e25aa2428b9f47a1d093d57cbf"
format_stamp: "Formatted at 2026-09-28 08:38:08 on dist-test-slave-w6c0"
server_key: "e9c77e49bf97e5aa644482e8aa63e431"
server_key_iv: "fc7002ebb097fa17703fd5e0ae75f2a9"
server_key_version: "encryptionkey@0"
I20260928 08:38:08.866335 8655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "90a1d5e25aa2428b9f47a1d093d57cbf"
format_stamp: "Formatted at 2026-09-28 08:38:08 on dist-test-slave-w6c0"
server_key: "e9c77e49bf97e5aa644482e8aa63e431"
server_key_iv: "fc7002ebb097fa17703fd5e0ae75f2a9"
server_key_version: "encryptionkey@0"
I20260928 08:38:08.871635 8655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:38:08.875458 8670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:08.876855 8655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:38:08.877033 8655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "90a1d5e25aa2428b9f47a1d093d57cbf"
format_stamp: "Formatted at 2026-09-28 08:38:08 on dist-test-slave-w6c0"
server_key: "e9c77e49bf97e5aa644482e8aa63e431"
server_key_iv: "fc7002ebb097fa17703fd5e0ae75f2a9"
server_key_version: "encryptionkey@0"
I20260928 08:38:08.877246 8655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:08.912321 8655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:08.916847 8655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:08.917092 8655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:08.936622 8655 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42985
I20260928 08:38:08.936796 8721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42985 every 8 connection(s)
I20260928 08:38:08.938056 8655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:38:08.941861 8722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:08.945637 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 8655
I20260928 08:38:08.945904 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:38:08.946734 388 external_mini_cluster.cc:1428] Setting key c3ed546395bdcf804e6ea8c28049ce1b
I20260928 08:38:08.953263 8722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf: Bootstrap starting.
I20260928 08:38:08.956434 8722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:08.957739 8722 log.cc:824] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf: Log is configured to *not* fsync() on all Append() calls
I20260928 08:38:08.961745 8722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf: No bootstrap required, opened a new log
Sep 28 08:38:08 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:38:08.971216 8722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a1d5e25aa2428b9f47a1d093d57cbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } }
I20260928 08:38:08.971694 8722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:08.971827 8722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a1d5e25aa2428b9f47a1d093d57cbf, State: Initialized, Role: FOLLOWER
I20260928 08:38:08.972760 8722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a1d5e25aa2428b9f47a1d093d57cbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } }
I20260928 08:38:08.973115 8722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:38:08.973280 8722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:38:08.973490 8722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:08.974666 8725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:08.949519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.0.97.62:42985)
0928 08:38:08.950815 (+ 1296us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:08.950830 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:08.950886 (+ 56us) server_negotiation.cc:407] Connection header received
0928 08:38:08.952685 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:08.952709 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:08.953152 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:08.953573 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:08.955229 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:08.956514 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:08.958456 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:08.959874 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:08.960089 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:08.965775 (+ 5686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:08.965784 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:08.965809 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:08.965877 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:08.969455 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:08.970273 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:08.970281 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:08.970290 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:08.970477 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:08.970903 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:08.970908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:08.970912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:08.971563 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:08.971962 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:08.972511 (+ 549us) server_negotiation.cc:299] Negotiation successful
0928 08:38:08.973105 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":502,"threads_started":1}
I20260928 08:38:08.976941 8722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a1d5e25aa2428b9f47a1d093d57cbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } }
I20260928 08:38:08.977798 8722 leader_election.cc:302] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90a1d5e25aa2428b9f47a1d093d57cbf; no voters:
I20260928 08:38:08.978873 8722 leader_election.cc:288] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:38:08.979009 8727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:38:08.979727 8727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [term 1 LEADER]: Becoming Leader. State: Replica: 90a1d5e25aa2428b9f47a1d093d57cbf, State: Running, Role: LEADER
I20260928 08:38:08.980284 8727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a1d5e25aa2428b9f47a1d093d57cbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } }
I20260928 08:38:08.981278 8722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:38:08.983412 8729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90a1d5e25aa2428b9f47a1d093d57cbf. Latest consensus state: current_term: 1 leader_uuid: "90a1d5e25aa2428b9f47a1d093d57cbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a1d5e25aa2428b9f47a1d093d57cbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } }
I20260928 08:38:08.983943 8729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [sys.catalog]: This master's current role is: LEADER
I20260928 08:38:08.986729 8728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90a1d5e25aa2428b9f47a1d093d57cbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a1d5e25aa2428b9f47a1d093d57cbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } }
I20260928 08:38:08.987097 8728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [sys.catalog]: This master's current role is: LEADER
I20260928 08:38:08.989737 8734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:38:08.995187 8734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:38:09.005044 8734 catalog_manager.cc:1347] Generated new cluster ID: 8f34a8c28381402eacde83d89bcea0db
I20260928 08:38:09.005177 8734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:38:09.012861 8734 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:38:09.014185 8734 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:38:09.029390 8734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf: Generated new TSK 0
I20260928 08:38:09.030268 8734 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:38:09.099283 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42985
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:38:09.465760 8748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:09.466028 8748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:09.466125 8748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:09.473634 8748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:09.473747 8748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:09.473865 8748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:38:09.483911 8748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:09.485145 8748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:09.486374 8748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:09.499758 8753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:09.499713 8754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:09.499715 8756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:09.500624 8748 server_base.cc:1034] running on GCE node
I20260928 08:38:09.501245 8748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:09.502260 8748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:09.503505 8748 hybrid_clock.cc:648] HybridClock initialized: now 1790584689503425 us; error 77 us; skew 500 ppm
I20260928 08:38:09.503854 8748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:38:09 dist-test-slave-w6c0 krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:38:09.507201 8748 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:38:09.508904 8748 webserver.cc:460] Webserver started at http://127.0.97.1:34585/ using document root <none> and password file <none>
I20260928 08:38:09.509620 8748 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:09.509773 8748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:09.510102 8748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:09.513052 8748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "bf5bcd8c2a274a56836c740cb6a6c28a"
format_stamp: "Formatted at 2026-09-28 08:38:09 on dist-test-slave-w6c0"
server_key: "46901e6f3565fac338865305f8aa6986"
server_key_iv: "c59c6a57b49ee6d424199b7ca23956e0"
server_key_version: "encryptionkey@0"
I20260928 08:38:09.513942 8748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bf5bcd8c2a274a56836c740cb6a6c28a"
format_stamp: "Formatted at 2026-09-28 08:38:09 on dist-test-slave-w6c0"
server_key: "46901e6f3565fac338865305f8aa6986"
server_key_iv: "c59c6a57b49ee6d424199b7ca23956e0"
server_key_version: "encryptionkey@0"
I20260928 08:38:09.519287 8748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 08:38:09.522806 8763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:09.524178 8748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 08:38:09.524382 8748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "bf5bcd8c2a274a56836c740cb6a6c28a"
format_stamp: "Formatted at 2026-09-28 08:38:09 on dist-test-slave-w6c0"
server_key: "46901e6f3565fac338865305f8aa6986"
server_key_iv: "c59c6a57b49ee6d424199b7ca23956e0"
server_key_version: "encryptionkey@0"
I20260928 08:38:09.524601 8748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:09.549504 8748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:09.554734 8748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:09.555114 8748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:09.556403 8748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:38:09.558074 8748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:38:09.558182 8748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:09.558353 8748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:38:09.558431 8748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:09.595264 8748 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37691
I20260928 08:38:09.595360 8875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37691 every 8 connection(s)
I20260928 08:38:09.596623 8748 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:38:09.600574 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 8748
I20260928 08:38:09.600878 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:38:09.601531 388 external_mini_cluster.cc:1428] Setting key 6cba34451f4fd0e912ac792fd28043ac
Sep 28 08:38:09 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:38:09.617599 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:09.599418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45209 (local address 127.0.97.62:42985)
0928 08:38:09.599889 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:09.599896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:09.600981 (+ 1085us) server_negotiation.cc:407] Connection header received
0928 08:38:09.603168 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:09.603173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:09.603263 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:09.603388 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:09.604882 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:09.605650 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:09.607321 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:09.607714 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:09.607862 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:09.611659 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:09.611665 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:09.611670 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:09.611702 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:09.614321 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:09.615242 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:09.615246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:09.615248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:09.615359 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:09.615858 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:09.615862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:09.615864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:09.616114 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:09.616242 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:09.617152 (+ 910us) server_negotiation.cc:299] Negotiation successful
0928 08:38:09.617335 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":224,"threads_started":1}
I20260928 08:38:09.619220 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:09.599989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.1:45209)
0928 08:38:09.600824 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:09.600870 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:09.602903 (+ 2033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:09.603620 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:09.603628 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:09.604136 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:09.604709 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:09.604727 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:09.605817 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:09.605827 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:09.607144 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:09.607156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:09.607965 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:09.607973 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:09.608208 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:09.608920 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:09.608942 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:09.611431 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:38:09.614451 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:09.614463 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:09.614483 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:09.615120 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:09.615473 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:09.615480 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:09.615487 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:09.615735 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:09.616275 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:38:09.616286 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:38:09.616740 (+ 454us) client_negotiation.cc:769] Sending connection context
0928 08:38:09.617053 (+ 313us) client_negotiation.cc:240] Negotiation successful
0928 08:38:09.617637 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":363,"threads_started":1}
I20260928 08:38:09.620836 8876 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42985
I20260928 08:38:09.621196 8876 heartbeater.cc:464] Registering TS with master...
I20260928 08:38:09.622166 8876 heartbeater.cc:511] Master 127.0.97.62:42985 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:38:09.624447 8687 ts_manager.cc:194] Registered new tserver with Master: bf5bcd8c2a274a56836c740cb6a6c28a (127.0.97.1:37691)
I20260928 08:38:09.625895 8687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:38:09.651973 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42985
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:38:09.979653 8882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:09.979943 8882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:09.980176 8882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:09.987713 8882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:09.987828 8882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:09.988026 8882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:38:09.997845 8882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:09.999838 8882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:10.001797 8882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:10.016196 8890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:10.016255 8887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:10.016255 8888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:10.017462 8882 server_base.cc:1034] running on GCE node
I20260928 08:38:10.018157 8882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:10.019158 8882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:10.020401 8882 hybrid_clock.cc:648] HybridClock initialized: now 1790584690020352 us; error 54 us; skew 500 ppm
I20260928 08:38:10.020716 8882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:38:10 dist-test-slave-w6c0 krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:38:10.023959 8882 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:38:10.025460 8882 webserver.cc:460] Webserver started at http://127.0.97.2:45565/ using document root <none> and password file <none>
I20260928 08:38:10.026160 8882 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:10.026309 8882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:10.026679 8882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:10.029677 8882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "2deefe73e9e14441b28f573eaa6fb586"
format_stamp: "Formatted at 2026-09-28 08:38:10 on dist-test-slave-w6c0"
server_key: "520fd512223613357c3c509381e0a5d9"
server_key_iv: "dacaf8452fd0288ab690aaa5745aaa80"
server_key_version: "encryptionkey@0"
I20260928 08:38:10.030441 8882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2deefe73e9e14441b28f573eaa6fb586"
format_stamp: "Formatted at 2026-09-28 08:38:10 on dist-test-slave-w6c0"
server_key: "520fd512223613357c3c509381e0a5d9"
server_key_iv: "dacaf8452fd0288ab690aaa5745aaa80"
server_key_version: "encryptionkey@0"
I20260928 08:38:10.035818 8882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:38:10.039279 8897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:10.040577 8882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:38:10.040756 8882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "2deefe73e9e14441b28f573eaa6fb586"
format_stamp: "Formatted at 2026-09-28 08:38:10 on dist-test-slave-w6c0"
server_key: "520fd512223613357c3c509381e0a5d9"
server_key_iv: "dacaf8452fd0288ab690aaa5745aaa80"
server_key_version: "encryptionkey@0"
I20260928 08:38:10.040972 8882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:10.057950 8882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:10.062358 8882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:10.062690 8882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:10.063651 8882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:38:10.064985 8882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:38:10.065088 8882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:10.065248 8882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:38:10.065320 8882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:10.098467 8882 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41255
I20260928 08:38:10.098553 9009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41255 every 8 connection(s)
I20260928 08:38:10.099798 8882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:38:10.103119 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 8882
I20260928 08:38:10.103287 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:38:10.103737 388 external_mini_cluster.cc:1428] Setting key 7825ff38081c391f56167ab9abca8ff3
Sep 28 08:38:10 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:38:10.120759 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:10.102828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50435 (local address 127.0.97.62:42985)
0928 08:38:10.103046 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:10.103054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:10.104351 (+ 1297us) server_negotiation.cc:407] Connection header received
0928 08:38:10.106377 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:10.106385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:10.106471 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:10.106601 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:10.108004 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:10.109100 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:10.110637 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:10.111388 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:10.111528 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:10.114961 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:10.114969 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:10.114973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:10.115017 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:10.117594 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:10.118333 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:10.118338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:10.118348 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:10.118468 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:10.118951 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:10.118956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:10.118959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:10.119235 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:10.119389 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:10.120190 (+ 801us) server_negotiation.cc:299] Negotiation successful
0928 08:38:10.120407 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:38:10.121830 9012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:10.103343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.2:50435)
0928 08:38:10.104185 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:10.104230 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:10.106118 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:10.106790 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:10.106798 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:10.107306 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:10.107839 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:10.107854 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:10.109256 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:10.109259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:10.110459 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:10.110469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:10.111584 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:10.111588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:10.111790 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:10.112488 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:10.112509 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:10.114706 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:38:10.117755 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:10.117764 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:10.117782 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:10.118203 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:10.118627 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:10.118634 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:10.118637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:10.118832 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:10.119363 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:38:10.119370 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:38:10.119756 (+ 386us) client_negotiation.cc:769] Sending connection context
0928 08:38:10.120095 (+ 339us) client_negotiation.cc:240] Negotiation successful
0928 08:38:10.120636 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":351,"threads_started":1}
I20260928 08:38:10.123422 9010 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42985
I20260928 08:38:10.123786 9010 heartbeater.cc:464] Registering TS with master...
I20260928 08:38:10.124763 9010 heartbeater.cc:511] Master 127.0.97.62:42985 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 08:38:10.126454 8687 ts_manager.cc:194] Registered new tserver with Master: 2deefe73e9e14441b28f573eaa6fb586 (127.0.97.2:41255)
I20260928 08:38:10.127381 8687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:38:10.153855 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42985
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:38:10.484059 9015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:10.484361 9015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:10.484527 9015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:10.491786 9015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:10.491931 9015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:10.492105 9015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:38:10.501595 9015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:10.502936 9015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:10.504236 9015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:10.516865 9021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:10.516928 9023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:10.516991 9020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:10.517820 9015 server_base.cc:1034] running on GCE node
I20260928 08:38:10.518460 9015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:10.519433 9015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:10.520663 9015 hybrid_clock.cc:648] HybridClock initialized: now 1790584690520618 us; error 38 us; skew 500 ppm
I20260928 08:38:10.520982 9015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:38:10 dist-test-slave-w6c0 krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:38:10.524114 9015 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:38:10.525910 9015 webserver.cc:460] Webserver started at http://127.0.97.3:35489/ using document root <none> and password file <none>
I20260928 08:38:10.526805 9015 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:10.526958 9015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:10.527271 9015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:10.530129 9015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "158014b49eea4b5b806f2ac22be7ec96"
format_stamp: "Formatted at 2026-09-28 08:38:10 on dist-test-slave-w6c0"
server_key: "7aa485501dd64c2f8178e911c8af9b81"
server_key_iv: "f66e27dceb5525ab88b7cdacfd3f69a0"
server_key_version: "encryptionkey@0"
I20260928 08:38:10.531025 9015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "158014b49eea4b5b806f2ac22be7ec96"
format_stamp: "Formatted at 2026-09-28 08:38:10 on dist-test-slave-w6c0"
server_key: "7aa485501dd64c2f8178e911c8af9b81"
server_key_iv: "f66e27dceb5525ab88b7cdacfd3f69a0"
server_key_version: "encryptionkey@0"
I20260928 08:38:10.536291 9015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 08:38:10.539698 9030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:10.541426 9015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:38:10.541723 9015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "158014b49eea4b5b806f2ac22be7ec96"
format_stamp: "Formatted at 2026-09-28 08:38:10 on dist-test-slave-w6c0"
server_key: "7aa485501dd64c2f8178e911c8af9b81"
server_key_iv: "f66e27dceb5525ab88b7cdacfd3f69a0"
server_key_version: "encryptionkey@0"
I20260928 08:38:10.542029 9015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:10.560461 9015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:10.564674 9015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:10.564987 9015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:10.565903 9015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:38:10.567284 9015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:38:10.567384 9015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:10.567538 9015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:38:10.567612 9015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:10.599601 9015 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42027
I20260928 08:38:10.599620 9142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42027 every 8 connection(s)
I20260928 08:38:10.601137 9015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:38:10.604419 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 9015
I20260928 08:38:10.604604 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:38:10.605178 388 external_mini_cluster.cc:1428] Setting key 508eaf7a37fc6605ab52c33be285b1ab
Sep 28 08:38:10 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:38:10.621341 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:10.603745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54057 (local address 127.0.97.62:42985)
0928 08:38:10.603953 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:10.603959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:10.605223 (+ 1264us) server_negotiation.cc:407] Connection header received
0928 08:38:10.607603 (+ 2380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:10.607609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:10.607691 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:10.607798 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:10.609329 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:10.610051 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:10.611676 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:10.612075 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:10.612184 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:10.615672 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:10.615677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:10.615680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:10.615710 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:10.618252 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:10.619084 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:10.619091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:10.619099 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:10.619233 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:10.619668 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:10.619679 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:10.619682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:10.619946 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:10.620092 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:10.620791 (+ 699us) server_negotiation.cc:299] Negotiation successful
0928 08:38:10.620996 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:38:10.622303 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:10.604242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.3:54057)
0928 08:38:10.605046 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:10.605101 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:10.607355 (+ 2254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:10.608023 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:10.608034 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:10.608650 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:10.609169 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:10.609185 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:10.610236 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:10.610242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:10.611526 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:10.611538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:10.612275 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:10.612280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:10.612456 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:10.613045 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:10.613070 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:10.615461 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:38:10.618415 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:10.618424 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:10.618440 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:10.618953 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:10.619344 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:10.619350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:10.619353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:10.619542 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:10.620083 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:38:10.620091 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:38:10.620420 (+ 329us) client_negotiation.cc:769] Sending connection context
0928 08:38:10.620688 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 08:38:10.621191 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":337,"threads_started":1}
I20260928 08:38:10.623692 9143 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42985
I20260928 08:38:10.623950 9143 heartbeater.cc:464] Registering TS with master...
I20260928 08:38:10.624584 9143 heartbeater.cc:511] Master 127.0.97.62:42985 requested a full tablet report, sending...
I20260928 08:38:10.625931 8687 ts_manager.cc:194] Registered new tserver with Master: 158014b49eea4b5b806f2ac22be7ec96 (127.0.97.3:42027)
I20260928 08:38:10.626756 8687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54057
I20260928 08:38:10.628824 8876 heartbeater.cc:503] Master 127.0.97.62:42985 was elected leader, sending a full tablet report...
I20260928 08:38:10.632369 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260928 08:38:10.766731 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 8748
Sep 28 08:38:10 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:38:10.782869 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:10.767992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38444 (local address 127.0.97.1:37691)
0928 08:38:10.768718 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:10.768725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:10.768752 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:38:10.768816 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:10.768822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:10.769024 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:10.769173 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:10.770462 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:10.771539 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:10.772789 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:10.773372 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:10.773483 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:10.777439 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:10.777450 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:10.777465 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:10.777500 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:10.780229 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:10.780841 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:10.780848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:10.780850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:10.780993 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:10.781313 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:10.781316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:10.781318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:10.781799 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:10.782120 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:10.782320 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 08:38:10.782580 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":418,"threads_started":1}
I20260928 08:38:10.784119 8871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:10.866346 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 8748
I20260928 08:38:10.883363 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 8882
Sep 28 08:38:10 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:38:10.897825 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:10.884182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54304 (local address 127.0.97.2:41255)
0928 08:38:10.884748 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:10.884755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:10.884780 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:38:10.884847 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:10.884854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:10.885054 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:10.885186 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:10.886263 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:10.887289 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:10.888448 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:10.888948 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:10.889040 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:10.893064 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:10.893074 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:10.893086 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:10.893115 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:10.895645 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:10.896201 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:10.896207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:10.896209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:10.896317 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:10.896603 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:10.896605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:10.896607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:10.896904 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:10.897125 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:10.897351 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 08:38:10.897539 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":293,"threads_started":1}
I20260928 08:38:10.899147 9005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:10.986860 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 8882
I20260928 08:38:11.002893 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 9015
Sep 28 08:38:11 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:38:11.020191 9157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:11.003941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41370 (local address 127.0.97.3:42027)
0928 08:38:11.004904 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:11.004911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:11.004942 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:38:11.004999 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:11.005005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:11.005229 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:11.005364 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:11.006682 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:11.007715 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:11.008971 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:11.009511 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:11.009610 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:11.014039 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:11.014050 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:11.014064 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:11.014101 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:11.017024 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:11.017860 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:11.017872 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:11.017874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:11.018139 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:11.018685 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:11.018693 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:11.018700 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:11.019111 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:11.019446 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:11.019685 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 08:38:11.019881 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":344,"threads_started":1}
I20260928 08:38:11.021500 9138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:11.104936 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 9015
I20260928 08:38:11.120890 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 8655
I20260928 08:38:11.122035 8717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:11.226025 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 8655
I20260928 08:38:11.247133 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42985
--webserver_interface=127.0.97.62
--webserver_port=42615
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:38:11.565104 9160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:11.565376 9160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:11.565465 9160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:11.572525 9160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:38:11.572625 9160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:11.572670 9160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:11.572727 9160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:38:11.572770 9160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:38:11.582278 9160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:11.583568 9160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:11.584779 9160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:11.596037 9165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:11.595942 9168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:11.595976 9166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:11.596827 9160 server_base.cc:1034] running on GCE node
I20260928 08:38:11.597685 9160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:11.599144 9160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:11.600349 9160 hybrid_clock.cc:648] HybridClock initialized: now 1790584691600300 us; error 52 us; skew 500 ppm
I20260928 08:38:11.600708 9160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:38:11 dist-test-slave-w6c0 krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:38:11.603868 9160 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:38:11.605228 9160 webserver.cc:460] Webserver started at http://127.0.97.62:42615/ using document root <none> and password file <none>
I20260928 08:38:11.605860 9160 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:11.606001 9160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:11.611343 9160 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:38:11.614022 9175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:11.615445 9160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:38:11.615586 9160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "90a1d5e25aa2428b9f47a1d093d57cbf"
format_stamp: "Formatted at 2026-09-28 08:38:08 on dist-test-slave-w6c0"
server_key: "e9c77e49bf97e5aa644482e8aa63e431"
server_key_iv: "fc7002ebb097fa17703fd5e0ae75f2a9"
server_key_version: "encryptionkey@0"
I20260928 08:38:11.616557 9160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:11.638482 9160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:11.642812 9160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:11.643021 9160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:11.661041 9160 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42985
I20260928 08:38:11.661079 9226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42985 every 8 connection(s)
I20260928 08:38:11.662570 9160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:38:11.667582 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 9160
I20260928 08:38:11.668524 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:37691
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42985
--webserver_port=34585
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:38:11.670130 9227 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:38:11.675179 9227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf: Bootstrap starting.
I20260928 08:38:11.693943 9227 log.cc:824] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf: Log is configured to *not* fsync() on all Append() calls
I20260928 08:38:11.702455 9227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:38:11.703016 9227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf: Bootstrap complete.
I20260928 08:38:11.711889 9227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a1d5e25aa2428b9f47a1d093d57cbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } }
I20260928 08:38:11.712813 9227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a1d5e25aa2428b9f47a1d093d57cbf, State: Initialized, Role: FOLLOWER
I20260928 08:38:11.713400 9227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a1d5e25aa2428b9f47a1d093d57cbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } }
I20260928 08:38:11.713641 9227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:38:11.713773 9227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:38:11.713954 9227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:38:11.717641 9227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a1d5e25aa2428b9f47a1d093d57cbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } }
I20260928 08:38:11.718122 9227 leader_election.cc:302] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90a1d5e25aa2428b9f47a1d093d57cbf; no voters:
I20260928 08:38:11.718820 9227 leader_election.cc:288] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 08:38:11.718963 9231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:38:11.720165 9231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [term 2 LEADER]: Becoming Leader. State: Replica: 90a1d5e25aa2428b9f47a1d093d57cbf, State: Running, Role: LEADER
I20260928 08:38:11.720788 9231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a1d5e25aa2428b9f47a1d093d57cbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } }
I20260928 08:38:11.721091 9227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:38:11.723316 9233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90a1d5e25aa2428b9f47a1d093d57cbf. Latest consensus state: current_term: 2 leader_uuid: "90a1d5e25aa2428b9f47a1d093d57cbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a1d5e25aa2428b9f47a1d093d57cbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } }
I20260928 08:38:11.723657 9233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [sys.catalog]: This master's current role is: LEADER
I20260928 08:38:11.724198 9232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "90a1d5e25aa2428b9f47a1d093d57cbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a1d5e25aa2428b9f47a1d093d57cbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42985 } } }
I20260928 08:38:11.724502 9232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf [sys.catalog]: This master's current role is: LEADER
I20260928 08:38:11.732743 9236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:38:11.739576 9236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:38:11.741359 9236 catalog_manager.cc:1259] Loaded cluster ID: 8f34a8c28381402eacde83d89bcea0db
I20260928 08:38:11.741508 9236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:38:11.747397 9236 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:38:11.749277 9236 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 90a1d5e25aa2428b9f47a1d093d57cbf: Loaded TSK: 0
I20260928 08:38:11.751209 9236 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 08:38:12.028483 9229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:12.028757 9229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:12.028901 9229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:12.036330 9229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:12.036458 9229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:12.036674 9229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:38:12.046715 9229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:37691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34585
--tserver_master_addrs=127.0.97.62:42985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:12.048049 9229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:12.049353 9229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:12.063643 9253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:12.063712 9254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:12.064527 9256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:12.064813 9229 server_base.cc:1034] running on GCE node
I20260928 08:38:12.065476 9229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:12.066494 9229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:12.067734 9229 hybrid_clock.cc:648] HybridClock initialized: now 1790584692067678 us; error 50 us; skew 500 ppm
I20260928 08:38:12.068058 9229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:38:12 dist-test-slave-w6c0 krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584692, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:38:12.071301 9229 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260928 08:38:12.073122 9229 webserver.cc:460] Webserver started at http://127.0.97.1:34585/ using document root <none> and password file <none>
I20260928 08:38:12.073807 9229 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:12.073951 9229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:12.079342 9229 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:38:12.082141 9263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:12.083550 9229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:38:12.083715 9229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "bf5bcd8c2a274a56836c740cb6a6c28a"
format_stamp: "Formatted at 2026-09-28 08:38:09 on dist-test-slave-w6c0"
server_key: "46901e6f3565fac338865305f8aa6986"
server_key_iv: "c59c6a57b49ee6d424199b7ca23956e0"
server_key_version: "encryptionkey@0"
I20260928 08:38:12.084623 9229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:12.102227 9229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:12.106436 9229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:12.106766 9229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:12.107872 9229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:38:12.109220 9229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:38:12.109330 9229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:12.109467 9229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:38:12.109567 9229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:12.142582 9229 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37691
I20260928 08:38:12.142747 9375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37691 every 8 connection(s)
I20260928 08:38:12.144034 9229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:38:12.153208 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 9229
I20260928 08:38:12.153954 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41255
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42985
--webserver_port=45565
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:38:12 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:12 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:12.163317 9379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:12.147072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.1:58651)
0928 08:38:12.147860 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:12.147901 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:12.149513 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:12.150750 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:12.150757 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:12.151247 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:12.151762 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:12.151778 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.153337 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.153344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:12.155536 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:12.155555 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.156679 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.156688 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:12.156985 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:12.157736 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:12.157769 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:12.162027 (+ 4258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":411,"threads_started":1}
W20260928 08:38:12.163342 9378 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:12.146969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58651 (local address 127.0.97.62:42985)
0928 08:38:12.147708 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:12.147723 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:12.148021 (+ 298us) server_negotiation.cc:407] Connection header received
0928 08:38:12.149973 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:12.149986 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:12.150297 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:12.150652 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:12.151932 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.153163 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.155708 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.156434 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.156571 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:12.162360 (+ 5789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58651: BlockingRecv error: recv got EOF from 127.0.97.1:58651 (error 108)
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":238,"threads_started":1}
W20260928 08:38:12.164137 9376 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42985 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:42985: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 28 08:38:12 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:12 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:12.172415 9379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:12.164751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.1:57843)
0928 08:38:12.164904 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:12.164922 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:12.165025 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:12.165353 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:12.165358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:12.165687 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:12.165820 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:12.165828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.166900 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.166906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:12.167792 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:12.167801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.168476 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.168482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:12.168652 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:12.169272 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:12.169289 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:12.172118 (+ 2829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 08:38:12.172536 9378 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:12.164928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57843 (local address 127.0.97.62:42985)
0928 08:38:12.165106 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:12.165113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:12.165129 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:38:12.165178 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:12.165181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:12.165230 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:12.165331 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:12.165962 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.166767 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.167925 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.168320 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.168447 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:12.172305 (+ 3858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57843: BlockingRecv error: recv got EOF from 127.0.97.1:57843 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 08:38:12 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:12 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:12.181923 9379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:12.173184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.1:42621)
0928 08:38:12.173419 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:12.173442 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:12.173552 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:12.173874 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:12.173882 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:12.174312 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:12.174579 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:12.174592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.175525 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.175532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:12.177290 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:12.177306 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.177956 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.177962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:12.178144 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:12.178813 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:12.178834 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:12.181700 (+ 2866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
W20260928 08:38:12.182246 9378 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:12.173418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42621 (local address 127.0.97.62:42985)
0928 08:38:12.173565 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:12.173569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:12.173583 (+ 14us) server_negotiation.cc:407] Connection header received
0928 08:38:12.173672 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:12.173675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:12.173735 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:12.173808 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:12.174714 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.175383 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.177427 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.177813 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.177903 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:12.181956 (+ 4053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42621: BlockingRecv error: recv got EOF from 127.0.97.1:42621 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 08:38:12.182286 9376 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:38:12.486816 9380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:12.487103 9380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:12.487315 9380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:12.494612 9380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:12.494717 9380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:12.494917 9380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:38:12.504712 9380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45565
--tserver_master_addrs=127.0.97.62:42985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:12.506058 9380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:12.507359 9380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:12.520674 9386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:12.520674 9388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:12.520771 9385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:12.521823 9380 server_base.cc:1034] running on GCE node
I20260928 08:38:12.522513 9380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:12.523480 9380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:12.524691 9380 hybrid_clock.cc:648] HybridClock initialized: now 1790584692524657 us; error 27 us; skew 500 ppm
I20260928 08:38:12.525017 9380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:38:12 dist-test-slave-w6c0 krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584692, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:38:12.528192 9380 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260928 08:38:12.529477 9380 webserver.cc:460] Webserver started at http://127.0.97.2:45565/ using document root <none> and password file <none>
I20260928 08:38:12.530117 9380 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:12.530221 9380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:12.535370 9380 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:38:12.537879 9395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:12.539177 9380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:38:12.539299 9380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "2deefe73e9e14441b28f573eaa6fb586"
format_stamp: "Formatted at 2026-09-28 08:38:10 on dist-test-slave-w6c0"
server_key: "520fd512223613357c3c509381e0a5d9"
server_key_iv: "dacaf8452fd0288ab690aaa5745aaa80"
server_key_version: "encryptionkey@0"
I20260928 08:38:12.540161 9380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:12.558903 9380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:12.562707 9380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:12.562920 9380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:12.563771 9380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:38:12.564947 9380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:38:12.565028 9380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:12.565109 9380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:38:12.565163 9380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:12.598122 9380 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41255
I20260928 08:38:12.598168 9507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41255 every 8 connection(s)
I20260928 08:38:12.599565 9380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:38:12.607971 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 9380
I20260928 08:38:12.608587 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42027
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42985
--webserver_port=35489
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:38:12 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:12 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:38:12.615994 9378 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:12.602125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56521 (local address 127.0.97.62:42985)
0928 08:38:12.602321 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:12.602327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:12.603465 (+ 1138us) server_negotiation.cc:407] Connection header received
0928 08:38:12.605168 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:12.605172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:12.605232 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:12.605316 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:12.606772 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.607475 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.609171 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.609958 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.610105 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:12.615737 (+ 5632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56521: BlockingRecv error: recv got EOF from 127.0.97.2:56521 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:38:12.616734 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:12.602612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.2:56521)
0928 08:38:12.603299 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:12.603348 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:12.604935 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:12.605512 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:12.605520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:12.605993 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:12.606595 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:12.606615 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.607692 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.607695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:12.609028 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:12.609040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.610183 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.610195 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:12.610430 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:12.611210 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:12.611232 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:12.615561 (+ 4329us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":267,"threads_started":1}
W20260928 08:38:12.617460 9508 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42985 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:42985: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 28 08:38:12 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:12 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:38:12.628091 9378 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:12.618126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56687 (local address 127.0.97.62:42985)
0928 08:38:12.618284 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:12.618288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:12.618360 (+ 72us) server_negotiation.cc:407] Connection header received
0928 08:38:12.618582 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:12.618585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:12.618640 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:12.618717 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:12.619591 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.620399 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.622352 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.622868 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.622961 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:12.627912 (+ 4951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56687: BlockingRecv error: recv got EOF from 127.0.97.2:56687 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 08:38:12.628115 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:12.618065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.2:56687)
0928 08:38:12.618275 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:12.618296 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:12.618421 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:12.618773 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:12.618780 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:12.619247 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:12.619448 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:12.619462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.620537 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.620545 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:12.622213 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:12.622229 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.623006 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.623015 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:12.623235 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:12.623867 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:12.623895 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:12.627732 (+ 3837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
Sep 28 08:38:12 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:12 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:12.636448 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:12.629008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.2:42027)
0928 08:38:12.629181 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:12.629201 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:12.629300 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:12.629615 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:12.629620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:12.629907 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:12.630047 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:12.630054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.631103 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.631108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:12.631990 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:12.632000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.632679 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.632684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:12.632853 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:12.633441 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:12.633457 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:12.636233 (+ 2776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 08:38:12.636677 9378 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:12.629127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42027 (local address 127.0.97.62:42985)
0928 08:38:12.629304 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:12.629310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:12.629327 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:38:12.629415 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:12.629420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:12.629490 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:12.629590 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:12.630181 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.630965 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.632127 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:12.632541 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:12.632670 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:12.636467 (+ 3797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42027: BlockingRecv error: recv got EOF from 127.0.97.2:42027 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 08:38:12.636857 9508 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:38:12.936126 9511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:12.936398 9511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:12.936494 9511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:12.943763 9511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:12.943877 9511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:12.943991 9511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:38:12.953583 9511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35489
--tserver_master_addrs=127.0.97.62:42985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:12.954972 9511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:12.956336 9511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:12.969403 9516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:12.969465 9517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:12.969473 9519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:12.970311 9511 server_base.cc:1034] running on GCE node
I20260928 08:38:12.970942 9511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:12.971792 9511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:12.973032 9511 hybrid_clock.cc:648] HybridClock initialized: now 1790584692972976 us; error 49 us; skew 500 ppm
I20260928 08:38:12.973340 9511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:38:12 dist-test-slave-w6c0 krb5kdc[8642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584692, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:38:12.977329 9511 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260928 08:38:12.978926 9511 webserver.cc:460] Webserver started at http://127.0.97.3:35489/ using document root <none> and password file <none>
I20260928 08:38:12.979638 9511 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:12.979753 9511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:12.985383 9511 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 08:38:12.988133 9526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:12.989555 9511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:38:12.989722 9511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "158014b49eea4b5b806f2ac22be7ec96"
format_stamp: "Formatted at 2026-09-28 08:38:10 on dist-test-slave-w6c0"
server_key: "7aa485501dd64c2f8178e911c8af9b81"
server_key_iv: "f66e27dceb5525ab88b7cdacfd3f69a0"
server_key_version: "encryptionkey@0"
I20260928 08:38:12.990729 9511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:13.008603 9511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:13.013940 9511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:13.014220 9511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:13.015311 9511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:38:13.016744 9511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:38:13.016898 9511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:13.017026 9511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:38:13.017120 9511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:13.050848 9511 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42027
I20260928 08:38:13.050931 9638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42027 every 8 connection(s)
I20260928 08:38:13.052285 9511 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:38:13.053733 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 9511
Sep 28 08:38:13 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:13 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:38:13.071702 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:13.055000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57165 (local address 127.0.97.62:42985)
0928 08:38:13.058370 (+ 3370us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:13.058378 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:13.058398 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:38:13.059280 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:13.059284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:13.059339 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:13.059420 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:13.061224 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.062112 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.063642 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.064240 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.064366 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:13.071345 (+ 6979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57165: BlockingRecv error: recv got EOF from 127.0.97.3:57165 (error 108)
Metrics: {"server-negotiator.queue_time_us":3245,"thread_start_us":1088,"threads_started":1}
I20260928 08:38:13.072661 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:13.055597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.3:57165)
0928 08:38:13.056714 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:13.056772 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:13.059016 (+ 2244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:13.059680 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:13.059692 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:13.060362 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:13.061064 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:13.061083 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.062256 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.062264 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:13.063502 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:13.063514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.064440 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.064445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:13.064640 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:13.065354 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:13.065401 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:13.071085 (+ 5684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":817,"thread_start_us":593,"threads_started":1}
W20260928 08:38:13.073385 9639 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42985 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:42985: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260928 08:38:13.075101 9378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:13.054797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.0.97.62:42985)
0928 08:38:13.055011 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:13.055017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:13.055709 (+ 692us) server_negotiation.cc:407] Connection header received
0928 08:38:13.055919 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:13.055928 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:13.056045 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:13.056181 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:13.058240 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.059066 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.060838 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.061328 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.061483 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:13.063979 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:13.063989 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:13.064007 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:13.064048 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:13.068190 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:13.070412 (+ 2222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:13.070424 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:13.070435 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:13.070677 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:13.071929 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:13.071934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:13.071937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:13.072635 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:13.072953 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:13.074354 (+ 1401us) server_negotiation.cc:299] Negotiation successful
0928 08:38:13.074758 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
Sep 28 08:38:13 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:13 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:38:13.084298 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:13.074010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37459 (local address 127.0.97.62:42985)
0928 08:38:13.074149 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:13.074154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:13.074195 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:38:13.074424 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:13.074431 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:13.074504 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:13.074774 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:13.075791 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.076955 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.078902 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.079648 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.079782 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:13.084143 (+ 4361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37459: BlockingRecv error: recv got EOF from 127.0.97.3:37459 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 08:38:13.084373 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:13.073956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.3:37459)
0928 08:38:13.074124 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:13.074144 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:13.074254 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:13.074952 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:13.074959 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:13.075416 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:13.075632 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:13.075646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.077092 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.077098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:13.078746 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:13.078761 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.079836 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.079846 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:13.080091 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:13.080847 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:13.080870 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:13.083994 (+ 3124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
Sep 28 08:38:13 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:13 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:13.093864 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:13.085002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.3:57949)
0928 08:38:13.085140 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:13.085157 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:13.085246 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:13.085640 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:13.085648 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:13.085997 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:13.086173 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:13.086187 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.087752 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.087756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:13.088858 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:13.088869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.089549 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.089555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:13.089727 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:13.090302 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:13.090319 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:13.093570 (+ 3251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 08:38:13.094146 9639 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:38:13.093998 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:13.085094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57949 (local address 127.0.97.62:42985)
0928 08:38:13.085330 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:13.085337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:13.085355 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:38:13.085399 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:13.085404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:13.085466 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:13.085563 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:13.086330 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.087597 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.088996 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.089413 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.089539 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:13.093735 (+ 4196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57949: BlockingRecv error: recv got EOF from 127.0.97.3:57949 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Sep 28 08:38:13 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:13 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:13.194913 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:13.183304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.1:40971)
0928 08:38:13.183913 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:13.183934 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:13.184146 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:13.184726 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:13.184730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:13.185472 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:13.185696 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:13.185710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.187042 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.187047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:13.188288 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:13.188302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.189512 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.189520 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:13.189730 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:13.190581 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:13.190609 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:13.194475 (+ 3866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":318,"threads_started":1}
W20260928 08:38:13.195909 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:13.183530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40971 (local address 127.0.97.62:42985)
0928 08:38:13.183791 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:13.183801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:13.184124 (+ 323us) server_negotiation.cc:407] Connection header received
0928 08:38:13.184277 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:13.184281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:13.184362 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:13.184458 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:13.185934 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.186824 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.188520 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.189262 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.189388 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:13.195124 (+ 5736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40971: BlockingRecv error: recv got EOF from 127.0.97.1:40971 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 28 08:38:13 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:13 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:13.646662 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:13.637473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.2:58919)
0928 08:38:13.637977 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:13.637995 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:13.638108 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:13.638429 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:13.638433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:13.638890 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:13.639082 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:13.639091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.640064 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.640068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:13.641565 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:13.641580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.642313 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.642318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:13.642471 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:13.643217 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:13.643235 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:13.646342 (+ 3107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":267,"threads_started":1}
W20260928 08:38:13.646876 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:13.637673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58919 (local address 127.0.97.62:42985)
0928 08:38:13.637885 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:13.637891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:13.638072 (+ 181us) server_negotiation.cc:407] Connection header received
0928 08:38:13.638231 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:13.638234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:13.638299 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:13.638380 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:13.639199 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.639943 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.641723 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:13.642117 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:13.642213 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:13.646620 (+ 4407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58919: BlockingRecv error: recv got EOF from 127.0.97.2:58919 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 08:38:14 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:14 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:14.111995 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:14.095506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.3:47303)
0928 08:38:14.096042 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:14.096065 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:14.096179 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:14.096575 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:14.096580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:14.097097 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:14.097391 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:14.097407 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:14.098625 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:14.098632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:14.099586 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:14.099597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:14.100320 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:14.100326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:14.100493 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:14.108070 (+ 7577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:14.108089 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:14.111520 (+ 3431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":313,"threads_started":1}
W20260928 08:38:14.112140 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:14.095472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47303 (local address 127.0.97.62:42985)
0928 08:38:14.095644 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:14.095649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:14.096143 (+ 494us) server_negotiation.cc:407] Connection header received
0928 08:38:14.096285 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:14.096289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:14.096348 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:14.096434 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:14.097555 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:14.098456 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:14.099740 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:14.100139 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:14.100280 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:14.111862 (+ 11582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47303: BlockingRecv error: recv got EOF from 127.0.97.3:47303 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 08:38:14 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:14 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:14.205895 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:14.196311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.1:49445)
0928 08:38:14.196879 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:14.196895 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:14.197052 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:14.197438 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:14.197441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:14.197826 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:14.197973 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:14.197980 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:14.199045 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:14.199049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:14.200171 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:14.200182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:14.200915 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:14.200919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:14.201024 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:14.201761 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:14.201787 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:14.205637 (+ 3850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":286,"threads_started":1}
W20260928 08:38:14.206212 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:14.196383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49445 (local address 127.0.97.62:42985)
0928 08:38:14.196586 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:14.196594 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:14.196976 (+ 382us) server_negotiation.cc:407] Connection header received
0928 08:38:14.197204 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:14.197209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:14.197293 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:14.197390 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:14.198133 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:14.198916 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:14.200324 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:14.200774 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:14.200874 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:14.205895 (+ 5021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49445: BlockingRecv error: recv got EOF from 127.0.97.1:49445 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 08:38:14 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:14 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:14.657191 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:14.647863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.2:59745)
0928 08:38:14.648725 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:14.648756 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:14.649006 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:14.649588 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:14.649593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:14.650036 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:14.650200 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:14.650209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:14.651319 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:14.651324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:14.652226 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:14.652238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:14.652960 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:14.652966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:14.653131 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:14.653800 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:14.653818 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:14.656806 (+ 2988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":352,"threads_started":1}
W20260928 08:38:14.657231 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:14.647940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59745 (local address 127.0.97.62:42985)
0928 08:38:14.648211 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:14.648222 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:14.648904 (+ 682us) server_negotiation.cc:407] Connection header received
0928 08:38:14.649286 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:14.649292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:14.649382 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:14.649509 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:14.650368 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:14.651172 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:14.652377 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:14.652793 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:14.652933 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:14.657019 (+ 4086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59745: BlockingRecv error: recv got EOF from 127.0.97.2:59745 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 08:38:15 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:15 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:15.123071 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:15.113241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.3:48215)
0928 08:38:15.114127 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:15.114153 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:15.114363 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:15.114841 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:15.114845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:15.115262 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:15.115431 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:15.115440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:15.116477 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:15.116480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:15.117389 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:15.117400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:15.118090 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:15.118095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:15.118202 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:15.118935 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:15.118954 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:15.122825 (+ 3871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":552,"threads_started":1}
W20260928 08:38:15.123309 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:15.113505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48215 (local address 127.0.97.62:42985)
0928 08:38:15.113943 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:15.113949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:15.114276 (+ 327us) server_negotiation.cc:407] Connection header received
0928 08:38:15.114520 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:15.114524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:15.114588 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:15.114685 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:15.115597 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:15.116355 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:15.117539 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:15.117948 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:15.118053 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:15.123028 (+ 4975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48215: BlockingRecv error: recv got EOF from 127.0.97.3:48215 (error 108)
Metrics: {"server-negotiator.queue_time_us":303}
Sep 28 08:38:15 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:15 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:15.215628 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:15.206974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.1:46713)
0928 08:38:15.207578 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:15.207594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:15.207747 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:15.208090 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:15.208102 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:15.208488 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:15.208651 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:15.208659 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:15.209674 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:15.209682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:15.210613 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:15.210624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:15.211333 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:15.211339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:15.211523 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:15.212229 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:15.212249 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:15.215371 (+ 3122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":303,"threads_started":1}
W20260928 08:38:15.215863 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:15.207116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46713 (local address 127.0.97.62:42985)
0928 08:38:15.207356 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:15.207364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:15.207683 (+ 319us) server_negotiation.cc:407] Connection header received
0928 08:38:15.207883 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:15.207888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:15.207958 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:15.208066 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:15.208804 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:15.209527 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:15.210763 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:15.211164 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:15.211304 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:15.215599 (+ 4295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46713: BlockingRecv error: recv got EOF from 127.0.97.1:46713 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 28 08:38:15 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:15 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:38:15.667906 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:15.658276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51987 (local address 127.0.97.62:42985)
0928 08:38:15.658432 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:15.658437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:15.658887 (+ 450us) server_negotiation.cc:407] Connection header received
0928 08:38:15.659026 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:15.659030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:15.659089 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:15.659170 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:15.659890 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:15.660967 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:15.662795 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:15.663232 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:15.663330 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:15.667724 (+ 4394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51987: BlockingRecv error: recv got EOF from 127.0.97.2:51987 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:38:15.667909 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:15.658208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.2:51987)
0928 08:38:15.658795 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:15.658818 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:15.658929 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:15.659217 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:15.659220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:15.659585 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:15.659746 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:15.659755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:15.661139 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:15.661144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:15.662658 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:15.662674 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:15.663367 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:15.663373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:15.663575 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:15.664228 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:15.664250 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:15.667563 (+ 3313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":366,"threads_started":1}
Sep 28 08:38:16 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:16 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:16.131421 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:16.123993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.3:39251)
0928 08:38:16.124493 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:16.124517 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:16.124626 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:16.124905 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:16.124908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:16.125232 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:16.125389 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:16.125397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:16.126328 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:16.126332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:16.127233 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:16.127243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:16.127853 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:16.127857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:16.127971 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:16.128539 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:16.128555 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:16.131183 (+ 2628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":279,"threads_started":1}
W20260928 08:38:16.131574 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:16.124061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39251 (local address 127.0.97.62:42985)
0928 08:38:16.124225 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:16.124230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:16.124599 (+ 369us) server_negotiation.cc:407] Connection header received
0928 08:38:16.124720 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:16.124724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:16.124782 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:16.124859 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:16.125513 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:16.126218 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:16.127352 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:16.127733 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:16.127825 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:16.131365 (+ 3540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39251: BlockingRecv error: recv got EOF from 127.0.97.3:39251 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 08:38:16 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:16 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:16.225314 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:16.216546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.1:45525)
0928 08:38:16.217060 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:16.217076 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:16.217225 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:16.217758 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:16.217763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:16.218174 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:16.218346 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:16.218354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:16.219640 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:16.219643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:16.220602 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:16.220613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:16.221390 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:16.221394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:16.221553 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:16.222195 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:16.222211 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:16.225098 (+ 2887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":284,"threads_started":1}
W20260928 08:38:16.225589 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:16.216612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45525 (local address 127.0.97.62:42985)
0928 08:38:16.216912 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:16.216920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:16.217160 (+ 240us) server_negotiation.cc:407] Connection header received
0928 08:38:16.217392 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:16.217398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:16.217465 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:16.217588 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:16.218554 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:16.219515 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:16.220761 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:16.221270 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:16.221358 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:16.225287 (+ 3929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45525: BlockingRecv error: recv got EOF from 127.0.97.1:45525 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Sep 28 08:38:16 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:16 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:16.676951 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:16.668868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.2:39609)
0928 08:38:16.669355 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:16.669376 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:16.669515 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:16.669839 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:16.669844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:16.670226 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:16.670364 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:16.670372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:16.671394 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:16.671401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:16.672342 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:16.672354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:16.673039 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:16.673045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:16.673242 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:16.673881 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:16.673898 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:16.676731 (+ 2833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":244,"threads_started":1}
W20260928 08:38:16.677145 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:16.668942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39609 (local address 127.0.97.62:42985)
0928 08:38:16.669182 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:16.669189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:16.669464 (+ 275us) server_negotiation.cc:407] Connection header received
0928 08:38:16.669638 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:16.669643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:16.669712 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:16.669815 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:16.670537 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:16.671261 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:16.672489 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:16.672888 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:16.673018 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:16.676906 (+ 3888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39609: BlockingRecv error: recv got EOF from 127.0.97.2:39609 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 08:38:17 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:17 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:17.140626 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:17.132400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.3:44009)
0928 08:38:17.132894 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:17.132909 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:17.133089 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:17.133447 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:17.133451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:17.133842 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:17.134011 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:17.134020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:17.135089 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:17.135096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:17.136029 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:17.136040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:17.136735 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:17.136740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:17.136928 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:17.137579 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:17.137596 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:17.140402 (+ 2806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":282,"threads_started":1}
W20260928 08:38:17.140841 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:17.132472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44009 (local address 127.0.97.62:42985)
0928 08:38:17.132742 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:17.132748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:17.133012 (+ 264us) server_negotiation.cc:407] Connection header received
0928 08:38:17.133240 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:17.133244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:17.133309 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:17.133417 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:17.134160 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:17.134954 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:17.136178 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:17.136583 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:17.136726 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:17.140603 (+ 3877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44009: BlockingRecv error: recv got EOF from 127.0.97.3:44009 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Sep 28 08:38:17 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:17 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:17.234174 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:17.226224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.1:39879)
0928 08:38:17.226842 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:17.226861 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:17.226982 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:17.227254 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:17.227257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:17.227630 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:17.227786 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:17.227793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:17.228799 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:17.228802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:17.229716 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:17.229728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:17.230378 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:17.230381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:17.230492 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:17.231129 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:17.231145 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:17.233936 (+ 2791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":308,"threads_started":1}
W20260928 08:38:17.234452 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:17.226276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39879 (local address 127.0.97.62:42985)
0928 08:38:17.226474 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:17.226480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:17.226953 (+ 473us) server_negotiation.cc:407] Connection header received
0928 08:38:17.227076 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:17.227079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:17.227137 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:17.227215 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:17.227938 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:17.228645 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:17.229854 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:17.230247 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:17.230340 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:17.234162 (+ 3822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39879: BlockingRecv error: recv got EOF from 127.0.97.1:39879 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 08:38:17 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:17 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:17.686710 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:17.677861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.2:58947)
0928 08:38:17.678395 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:17.678411 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:17.678583 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:17.678944 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:17.678948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:17.679329 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:17.679510 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:17.679519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:17.680690 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:17.680694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:17.681605 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:17.681616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:17.682377 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:17.682383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:17.682608 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:17.683387 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:17.683411 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:17.686385 (+ 2974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":278,"threads_started":1}
W20260928 08:38:17.686849 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:17.678029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58947 (local address 127.0.97.62:42985)
0928 08:38:17.678199 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:17.678204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:17.678491 (+ 287us) server_negotiation.cc:407] Connection header received
0928 08:38:17.678688 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:17.678692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:17.678750 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:17.678843 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:17.679632 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:17.680399 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:17.681749 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:17.682184 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:17.682282 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:17.686642 (+ 4360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58947: BlockingRecv error: recv got EOF from 127.0.97.2:58947 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 08:38:18 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:18 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:18.150810 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:18.141700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.3:38627)
0928 08:38:18.142233 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:18.142248 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:18.142432 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:18.142890 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:18.142894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:18.143470 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:18.143657 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:18.143669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:18.144976 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:18.144982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:18.146242 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:18.146253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:18.147063 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:18.147067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:18.147181 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:18.147818 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:18.147836 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:18.150566 (+ 2730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":277,"threads_started":1}
W20260928 08:38:18.150892 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:18.141794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38627 (local address 127.0.97.62:42985)
0928 08:38:18.142049 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:18.142056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:18.142344 (+ 288us) server_negotiation.cc:407] Connection header received
0928 08:38:18.142617 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:18.142622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:18.142694 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:18.142794 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:18.143835 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:18.144835 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:18.146362 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:18.146936 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:18.147028 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:18.150710 (+ 3682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38627: BlockingRecv error: recv got EOF from 127.0.97.3:38627 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 28 08:38:18 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:18 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:18.244287 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:18.235229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.1:37463)
0928 08:38:18.235730 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:18.235749 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:18.235868 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:18.236251 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:18.236254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:18.236647 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:18.236788 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:18.236817 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:18.238043 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:18.238048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:18.239482 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:18.239494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:18.240357 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:18.240362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:18.240491 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:18.241083 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:18.241100 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:18.244073 (+ 2973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":291,"threads_started":1}
W20260928 08:38:18.244693 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:18.235334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37463 (local address 127.0.97.62:42985)
0928 08:38:18.235612 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:18.235618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:18.235835 (+ 217us) server_negotiation.cc:407] Connection header received
0928 08:38:18.236020 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:18.236024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:18.236085 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:18.236171 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:18.236986 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:18.237850 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:18.239677 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:18.240117 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:18.240220 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:18.244252 (+ 4032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37463: BlockingRecv error: recv got EOF from 127.0.97.1:37463 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 28 08:38:18 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:18 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:18.696482 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:18.687703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.2:59823)
0928 08:38:18.688540 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:18.688556 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:18.688666 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:18.689148 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:18.689152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:18.689636 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:18.689787 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:18.689796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:18.690787 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:18.690791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:18.691671 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:18.691682 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:18.692350 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:18.692354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:18.692485 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:18.693137 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:18.693155 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:18.696142 (+ 2987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":600,"threads_started":1}
W20260928 08:38:18.696561 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:18.687772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59823 (local address 127.0.97.62:42985)
0928 08:38:18.687955 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:18.687959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:18.688646 (+ 687us) server_negotiation.cc:407] Connection header received
0928 08:38:18.688870 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:18.688876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:18.688987 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:18.689103 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:18.689932 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:18.690667 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:18.691798 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:18.692222 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:18.692312 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:18.696336 (+ 4024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59823: BlockingRecv error: recv got EOF from 127.0.97.2:59823 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 08:38:19 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:19 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:19.166100 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:19.151795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.3:36661)
0928 08:38:19.152332 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:19.152349 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:19.152457 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:19.152744 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:19.152747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:19.153106 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:19.153243 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:19.153250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:19.154786 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:19.154793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:19.155842 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:19.155853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:19.156536 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:19.156540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:19.156650 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:19.162852 (+ 6202us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:19.162870 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:19.165814 (+ 2944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":324,"threads_started":1}
W20260928 08:38:19.166293 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:19.151908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36661 (local address 127.0.97.62:42985)
0928 08:38:19.152070 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:19.152074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:19.152429 (+ 355us) server_negotiation.cc:407] Connection header received
0928 08:38:19.152560 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:19.152564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:19.152620 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:19.152700 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:19.153371 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:19.154608 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:19.155997 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:19.156404 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:19.156498 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:19.166025 (+ 9527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36661: BlockingRecv error: recv got EOF from 127.0.97.3:36661 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 08:38:19 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:19 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:19.254019 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:19.245427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.1:35247)
0928 08:38:19.246040 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:19.246056 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:19.246178 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:19.246589 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:19.246592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:19.246980 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:19.247148 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:19.247156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:19.248227 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:19.248233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:19.249396 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:19.249408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:19.250141 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:19.250146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:19.250309 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:19.250975 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:19.250992 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:19.253657 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":308,"threads_started":1}
W20260928 08:38:19.254065 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:19.245481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35247 (local address 127.0.97.62:42985)
0928 08:38:19.245689 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:19.245693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:19.246143 (+ 450us) server_negotiation.cc:407] Connection header received
0928 08:38:19.246334 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:19.246338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:19.246400 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:19.246488 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:19.247271 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:19.248037 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:19.249555 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:19.249986 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:19.250129 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:19.253836 (+ 3707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35247: BlockingRecv error: recv got EOF from 127.0.97.1:35247 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 28 08:38:19 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:19 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:19.706219 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:19.697536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.2:40817)
0928 08:38:19.698156 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:19.698172 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:19.698290 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:19.698607 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:19.698611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:19.699000 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:19.699170 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:19.699178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:19.700164 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:19.700168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:19.701316 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:19.701331 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:19.702317 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:19.702324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:19.702489 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:19.703231 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:19.703255 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:19.705974 (+ 2719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":279,"threads_started":1}
W20260928 08:38:19.706373 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:19.697628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40817 (local address 127.0.97.62:42985)
0928 08:38:19.697843 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:19.697848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:19.698253 (+ 405us) server_negotiation.cc:407] Connection header received
0928 08:38:19.698393 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:19.698397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:19.698456 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:19.698565 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:19.699311 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:19.700007 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:19.701484 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:19.702140 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:19.702266 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:19.706156 (+ 3890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40817: BlockingRecv error: recv got EOF from 127.0.97.2:40817 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 28 08:38:20 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:20 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:20.177947 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:20.167378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.3:33955)
0928 08:38:20.168129 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:20.168146 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:20.168273 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:20.168641 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:20.168644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:20.169050 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:20.169220 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:20.169229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:20.170407 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:20.170414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:20.171678 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:20.171700 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:20.172780 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:20.172789 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:20.172988 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:20.173993 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:20.174015 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:20.177564 (+ 3549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":416,"threads_started":1}
W20260928 08:38:20.178009 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:20.167418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33955 (local address 127.0.97.62:42985)
0928 08:38:20.167588 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:20.167694 (+ 106us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:20.168275 (+ 581us) server_negotiation.cc:407] Connection header received
0928 08:38:20.168395 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:20.168399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:20.168462 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:20.168553 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:20.169405 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:20.170149 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:20.171921 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:20.172553 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:20.172681 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:20.177755 (+ 5074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33955: BlockingRecv error: recv got EOF from 127.0.97.3:33955 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 08:38:20 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:20 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:20.265512 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:20.255381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.1:39357)
0928 08:38:20.255925 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:20.255942 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:20.256088 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:20.256398 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:20.256401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:20.256795 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:20.256944 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:20.256952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:20.258024 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:20.258028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:20.259265 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:20.259276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:20.260234 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:20.260241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:20.260411 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:20.261282 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:20.261299 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:20.265197 (+ 3898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":302,"threads_started":1}
W20260928 08:38:20.265726 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:20.255681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39357 (local address 127.0.97.62:42985)
0928 08:38:20.255931 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:20.255937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:20.256078 (+ 141us) server_negotiation.cc:407] Connection header received
0928 08:38:20.256193 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:20.256197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:20.256264 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:20.256351 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:20.257096 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:20.257861 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:20.259438 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:20.260021 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:20.260151 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:20.265445 (+ 5294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39357: BlockingRecv error: recv got EOF from 127.0.97.1:39357 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 08:38:20 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:20 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:20.715847 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:20.707268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.2:41679)
0928 08:38:20.707746 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:20.707762 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:20.707877 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:20.708208 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:20.708213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:20.708739 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:20.708958 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:20.708971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:20.710420 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:20.710424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:20.711365 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:20.711377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:20.712090 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:20.712094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:20.712219 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:20.712792 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:20.712809 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:20.715623 (+ 2814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":262,"threads_started":1}
W20260928 08:38:20.716146 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:20.707458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41679 (local address 127.0.97.62:42985)
0928 08:38:20.707704 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:20.707711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:20.707843 (+ 132us) server_negotiation.cc:407] Connection header received
0928 08:38:20.707992 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:20.707996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:20.708056 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:20.708134 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:20.709110 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:20.710268 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:20.711516 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:20.711937 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:20.712035 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:20.715869 (+ 3834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41679: BlockingRecv error: recv got EOF from 127.0.97.2:41679 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 28 08:38:21 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:21 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:21.190375 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:21.179158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.3:40415)
0928 08:38:21.182553 (+ 3395us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:21.182572 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:21.182703 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:21.182965 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:21.182968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:21.183330 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:21.183566 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:21.183577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:21.184968 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:21.184972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:21.185968 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:21.185980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:21.186690 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:21.186695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:21.186808 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:21.187375 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:21.187391 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:21.190141 (+ 2750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3262,"thread_start_us":3138,"threads_started":1}
W20260928 08:38:21.190577 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:21.179274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40415 (local address 127.0.97.62:42985)
0928 08:38:21.179500 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:21.179506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:21.182653 (+ 3147us) server_negotiation.cc:407] Connection header received
0928 08:38:21.182797 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:21.182801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:21.182865 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:21.182953 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:21.183712 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:21.184835 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:21.186097 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:21.186485 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:21.186650 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:21.190337 (+ 3687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40415: BlockingRecv error: recv got EOF from 127.0.97.3:40415 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 08:38:21 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:21 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:21.275415 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:21.266674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.1:48947)
0928 08:38:21.267230 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:21.267246 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:21.267356 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:21.267739 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:21.267743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:21.268287 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:21.268462 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:21.268471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:21.269511 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:21.269517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:21.270434 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:21.270444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:21.271208 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:21.271214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:21.271392 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:21.271998 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:21.272015 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:21.274968 (+ 2953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":314,"threads_started":1}
W20260928 08:38:21.275547 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:21.266810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48947 (local address 127.0.97.62:42985)
0928 08:38:21.266995 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:21.266999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:21.267340 (+ 341us) server_negotiation.cc:407] Connection header received
0928 08:38:21.267457 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:21.267461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:21.267563 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:21.267650 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:21.268628 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:21.269368 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:21.270639 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:21.271049 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:21.271204 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:21.275302 (+ 4098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48947: BlockingRecv error: recv got EOF from 127.0.97.1:48947 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 08:38:21 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:21 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:21.725324 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:21.716801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.2:56695)
0928 08:38:21.717473 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:21.717490 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:21.717607 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:21.717888 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:21.717891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:21.718287 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:21.718458 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:21.718467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:21.719556 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:21.719561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:21.720665 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:21.720679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:21.721536 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:21.721541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:21.721675 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:21.722346 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:21.722364 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:21.725066 (+ 2702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":392,"threads_started":1}
W20260928 08:38:21.725466 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:21.716886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56695 (local address 127.0.97.62:42985)
0928 08:38:21.717102 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:21.717113 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:21.717570 (+ 457us) server_negotiation.cc:407] Connection header received
0928 08:38:21.717706 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:21.717709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:21.717770 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:21.717872 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:21.718657 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:21.719409 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:21.720798 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:21.721362 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:21.721458 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:21.725254 (+ 3796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56695: BlockingRecv error: recv got EOF from 127.0.97.2:56695 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 28 08:38:22 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:22 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:22.204171 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:22.191373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.3:46575)
0928 08:38:22.191902 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:22.191917 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:22.192081 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:22.192447 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:22.192451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:22.192978 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:22.193206 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:22.193218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:22.195154 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:22.195159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:22.196942 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:22.196961 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:22.198007 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:22.198014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:22.198253 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:22.198973 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:22.198995 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:22.203739 (+ 4744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":294,"threads_started":1}
W20260928 08:38:22.204293 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:22.191440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46575 (local address 127.0.97.62:42985)
0928 08:38:22.191713 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:22.191721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:22.192006 (+ 285us) server_negotiation.cc:407] Connection header received
0928 08:38:22.192218 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:22.192223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:22.192306 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:22.192415 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:22.193320 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:22.194457 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:22.197108 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:22.197819 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:22.197960 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:22.204048 (+ 6088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46575: BlockingRecv error: recv got EOF from 127.0.97.3:46575 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 28 08:38:22 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:22 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:22.285243 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:22.276355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.1:43023)
0928 08:38:22.276924 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:22.276940 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:22.277107 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:22.277476 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:22.277481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:22.277857 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:22.278018 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:22.278026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:22.279321 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:22.279329 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:22.280472 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:22.280484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:22.281235 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:22.281241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:22.281411 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:22.282070 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:22.282088 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:22.285010 (+ 2922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":322,"threads_started":1}
W20260928 08:38:22.285495 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:22.276514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43023 (local address 127.0.97.62:42985)
0928 08:38:22.276763 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:22.276770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:22.277066 (+ 296us) server_negotiation.cc:407] Connection header received
0928 08:38:22.277250 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:22.277255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:22.277338 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:22.277460 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:22.278183 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:22.279191 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:22.280614 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:22.281073 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:22.281218 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:22.285204 (+ 3986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43023: BlockingRecv error: recv got EOF from 127.0.97.1:43023 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 28 08:38:22 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:22 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:22.735328 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:22.726435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.2:49985)
0928 08:38:22.726970 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:22.726991 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:22.727110 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:22.727432 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:22.727435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:22.727800 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:22.727940 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:22.727948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:22.728964 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:22.728968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:22.730187 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:22.730200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:22.731074 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:22.731080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:22.731249 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:22.732038 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:22.732056 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:22.735000 (+ 2944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":347,"threads_started":1}
W20260928 08:38:22.735425 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:22.726590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49985 (local address 127.0.97.62:42985)
0928 08:38:22.726807 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:22.726814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:22.727105 (+ 291us) server_negotiation.cc:407] Connection header received
0928 08:38:22.727228 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:22.727232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:22.727294 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:22.727385 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:22.728087 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:22.728774 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:22.730349 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:22.730891 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:22.731033 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:22.735165 (+ 4132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49985: BlockingRecv error: recv got EOF from 127.0.97.2:49985 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 08:38:23 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:23 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:23.214437 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:23.205773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.3:33471)
0928 08:38:23.206319 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:23.206334 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:23.206486 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:23.206912 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:23.206917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:23.207314 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:23.207504 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:23.207521 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:23.208600 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:23.208604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:23.209513 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:23.209536 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:23.210267 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:23.210271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:23.210396 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:23.211071 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:23.211087 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:23.214120 (+ 3033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":326,"threads_started":1}
W20260928 08:38:23.214706 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:23.206032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33471 (local address 127.0.97.62:42985)
0928 08:38:23.206254 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:23.206261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:23.206416 (+ 155us) server_negotiation.cc:407] Connection header received
0928 08:38:23.206698 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:23.206703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:23.206773 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:23.206882 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:23.207705 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:23.208463 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:23.209691 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:23.210111 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:23.210217 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:23.214322 (+ 4105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33471: BlockingRecv error: recv got EOF from 127.0.97.3:33471 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 28 08:38:23 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:23 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:23.294869 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:23.286253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.1:37833)
0928 08:38:23.286829 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:23.286845 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:23.286959 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:23.287313 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:23.287318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:23.287887 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:23.288074 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:23.288082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:23.289416 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:23.289420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:23.290322 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:23.290332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:23.291085 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:23.291089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:23.291200 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:23.291774 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:23.291790 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:23.294578 (+ 2788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":348,"threads_started":1}
W20260928 08:38:23.294952 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:23.286419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37833 (local address 127.0.97.62:42985)
0928 08:38:23.286635 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:23.286639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:23.286921 (+ 282us) server_negotiation.cc:407] Connection header received
0928 08:38:23.287079 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:23.287083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:23.287158 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:23.287244 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:23.288232 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:23.289268 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:23.290454 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:23.290951 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:23.291044 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:23.294762 (+ 3718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37833: BlockingRecv error: recv got EOF from 127.0.97.1:37833 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 28 08:38:23 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:23 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:23.745362 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:23.736349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.2:35209)
0928 08:38:23.736875 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:23.736891 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:23.737028 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:23.737377 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:23.737381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:23.737837 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:23.738016 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:23.738025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:23.739359 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:23.739364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:23.740298 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:23.740311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:23.741080 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:23.741085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:23.741211 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:23.741893 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:23.741912 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:23.745106 (+ 3194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":301,"threads_started":1}
W20260928 08:38:23.745587 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:23.736529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35209 (local address 127.0.97.62:42985)
0928 08:38:23.736693 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:23.736697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:23.737007 (+ 310us) server_negotiation.cc:407] Connection header received
0928 08:38:23.737176 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:23.737180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:23.737241 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:23.737326 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:23.738170 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:23.739175 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:23.740450 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:23.740900 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:23.740999 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:23.745335 (+ 4336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35209: BlockingRecv error: recv got EOF from 127.0.97.2:35209 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 08:38:24 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:24 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:24.230340 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:24.215515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.3:36273)
0928 08:38:24.216024 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:24.216040 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:24.216200 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:24.216549 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:24.216554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:24.216976 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:24.217136 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:24.217144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:24.218117 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:24.218125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:24.219076 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:24.219093 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:24.219865 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:24.219871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:24.220051 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:24.226994 (+ 6943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:24.227025 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:24.229993 (+ 2968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":288,"threads_started":1}
W20260928 08:38:24.230473 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:24.215578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36273 (local address 127.0.97.62:42985)
0928 08:38:24.215826 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:24.215835 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:24.216133 (+ 298us) server_negotiation.cc:407] Connection header received
0928 08:38:24.216333 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:24.216339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:24.216412 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:24.216518 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:24.217284 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:24.217991 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:24.219231 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:24.219712 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:24.219852 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:24.230241 (+ 10389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36273: BlockingRecv error: recv got EOF from 127.0.97.3:36273 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 08:38:24 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:24 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:24.304527 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:24.295968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.1:53825)
0928 08:38:24.296465 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:24.296482 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:24.296602 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:24.297107 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:24.297110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:24.297616 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:24.297769 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:24.297777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:24.299219 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:24.299223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:24.300127 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:24.300137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:24.300824 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:24.300828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:24.300950 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:24.301613 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:24.301631 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:24.304287 (+ 2656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":273,"threads_started":1}
W20260928 08:38:24.304947 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:24.296026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53825 (local address 127.0.97.62:42985)
0928 08:38:24.296300 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:24.296307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:24.296568 (+ 261us) server_negotiation.cc:407] Connection header received
0928 08:38:24.296810 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:24.296818 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:24.296947 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:24.297059 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:24.297928 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:24.299095 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:24.300259 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:24.300680 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:24.300799 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:24.304485 (+ 3686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53825: BlockingRecv error: recv got EOF from 127.0.97.1:53825 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 08:38:24 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:24 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:24.754451 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:24.746376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.2:60521)
0928 08:38:24.746918 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:24.746942 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:24.747091 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:24.747451 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:24.747456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:24.747862 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:24.748011 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:24.748019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:24.749014 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:24.749020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:24.749919 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:24.749929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:24.750682 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:24.750687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:24.750870 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:24.751455 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:24.751473 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:24.754109 (+ 2636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":291,"threads_started":1}
W20260928 08:38:24.754547 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:24.746435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60521 (local address 127.0.97.62:42985)
0928 08:38:24.746697 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:24.746704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:24.747031 (+ 327us) server_negotiation.cc:407] Connection header received
0928 08:38:24.747223 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:24.747231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:24.747304 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:24.747410 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:24.748151 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:24.748870 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:24.750059 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:24.750486 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:24.750673 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:24.754291 (+ 3618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60521: BlockingRecv error: recv got EOF from 127.0.97.2:60521 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 28 08:38:25 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:25 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:25.239555 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:25.231413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.3:55323)
0928 08:38:25.231935 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:25.231951 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:25.232100 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:25.232523 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:25.232532 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:25.232951 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:25.233102 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:25.233111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:25.234084 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:25.234088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:25.235005 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:25.235017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:25.235645 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:25.235649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:25.235789 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:25.236492 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:25.236509 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:25.239322 (+ 2813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":287,"threads_started":1}
W20260928 08:38:25.239878 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:25.231453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55323 (local address 127.0.97.62:42985)
0928 08:38:25.231628 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:25.231632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:25.232042 (+ 410us) server_negotiation.cc:407] Connection header received
0928 08:38:25.232234 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:25.232239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:25.232343 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:25.232438 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:25.233229 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:25.233961 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:25.235129 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:25.235520 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:25.235610 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:25.239574 (+ 3964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55323: BlockingRecv error: recv got EOF from 127.0.97.3:55323 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 08:38:25 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:25 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:25.313793 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:25.305458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.1:58613)
0928 08:38:25.306005 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:25.306021 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:25.306189 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:25.306569 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:25.306573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:25.307002 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:25.307146 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:25.307154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:25.308179 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:25.308186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:25.309110 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:25.309121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:25.309840 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:25.309846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:25.310006 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:25.310667 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:25.310683 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:25.313569 (+ 2886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":299,"threads_started":1}
W20260928 08:38:25.313982 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:25.305548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58613 (local address 127.0.97.62:42985)
0928 08:38:25.305813 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:25.305822 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:25.306120 (+ 298us) server_negotiation.cc:407] Connection header received
0928 08:38:25.306319 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:25.306326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:25.306403 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:25.306515 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:25.307303 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:25.308024 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:25.309270 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:25.309653 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:25.309793 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:25.313763 (+ 3970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58613: BlockingRecv error: recv got EOF from 127.0.97.1:58613 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 28 08:38:25 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:25 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:25.764390 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:25.755560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.2:44865)
0928 08:38:25.756125 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:25.756140 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:25.756299 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:25.756647 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:25.756651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:25.757144 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:25.757300 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:25.757309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:25.758359 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:25.758366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:25.759319 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:25.759331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:25.760177 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:25.760183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:25.760356 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:25.761047 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:25.761065 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:25.764151 (+ 3086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":292,"threads_started":1}
W20260928 08:38:25.764590 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:25.755714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44865 (local address 127.0.97.62:42985)
0928 08:38:25.755954 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:25.755962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:25.756232 (+ 270us) server_negotiation.cc:407] Connection header received
0928 08:38:25.756428 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:25.756434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:25.756509 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:25.756617 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:25.757448 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:25.758200 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:25.759474 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:25.760010 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:25.760140 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:25.764349 (+ 4209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44865: BlockingRecv error: recv got EOF from 127.0.97.2:44865 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 08:38:26 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:26 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:26.248752 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:26.240468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.3:37861)
0928 08:38:26.241004 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:26.241020 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:26.241191 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:26.241533 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:26.241537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:26.241948 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:26.242113 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:26.242121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:26.243235 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:26.243239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:26.244306 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:26.244318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:26.245058 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:26.245063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:26.245173 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:26.245766 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:26.245782 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:26.248520 (+ 2738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":293,"threads_started":1}
W20260928 08:38:26.248935 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:26.240550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37861 (local address 127.0.97.62:42985)
0928 08:38:26.240813 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:26.240822 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:26.241122 (+ 300us) server_negotiation.cc:407] Connection header received
0928 08:38:26.241324 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:26.241330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:26.241397 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:26.241504 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:26.242260 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:26.243054 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:26.244440 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:26.244866 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:26.244956 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:26.248740 (+ 3784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37861: BlockingRecv error: recv got EOF from 127.0.97.3:37861 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 28 08:38:26 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:26 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:26.323658 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:26.314869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.1:57787)
0928 08:38:26.315371 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:26.315386 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:26.315542 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:26.315908 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:26.315912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:26.316328 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:26.316497 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:26.316505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:26.317506 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:26.317511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:26.318425 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:26.318436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:26.319224 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:26.319230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:26.319408 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:26.320045 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:26.320061 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:26.323302 (+ 3241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":273,"threads_started":1}
W20260928 08:38:26.323725 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:26.314938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57787 (local address 127.0.97.62:42985)
0928 08:38:26.315238 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:26.315245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:26.315470 (+ 225us) server_negotiation.cc:407] Connection header received
0928 08:38:26.315663 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:26.315669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:26.315766 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:26.315877 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:26.316639 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:26.317355 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:26.318596 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:26.319067 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:26.319202 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:26.323481 (+ 4279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57787: BlockingRecv error: recv got EOF from 127.0.97.1:57787 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
Sep 28 08:38:26 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:26 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:26.774379 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:26.765342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.2:50503)
0928 08:38:26.765938 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:26.765954 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:26.766135 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:26.766479 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:26.766483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:26.766901 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:26.767083 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:26.767092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:26.768213 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:26.768219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:26.769623 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:26.769639 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:26.770353 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:26.770357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:26.770472 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:26.771169 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:26.771186 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:26.774007 (+ 2821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":357,"threads_started":1}
W20260928 08:38:26.774435 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:26.765492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50503 (local address 127.0.97.62:42985)
0928 08:38:26.765775 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:26.765783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:26.766051 (+ 268us) server_negotiation.cc:407] Connection header received
0928 08:38:26.766269 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:26.766275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:26.766340 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:26.766448 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:26.767235 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:26.768052 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:26.769791 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:26.770223 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:26.770322 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:26.774205 (+ 3883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50503: BlockingRecv error: recv got EOF from 127.0.97.2:50503 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Sep 28 08:38:27 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:27 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:27.258030 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:27.249859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.3:41263)
0928 08:38:27.250377 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:27.250398 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:27.250568 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:27.250935 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:27.250940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:27.251446 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:27.251673 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:27.251683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:27.252638 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:27.252641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:27.253529 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:27.253539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:27.254186 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:27.254190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:27.254304 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:27.254960 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:27.254984 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:27.257761 (+ 2777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":260,"threads_started":1}
W20260928 08:38:27.258210 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:27.249997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41263 (local address 127.0.97.62:42985)
0928 08:38:27.250243 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:27.250249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:27.250494 (+ 245us) server_negotiation.cc:407] Connection header received
0928 08:38:27.250728 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:27.250732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:27.250808 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:27.250908 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:27.251797 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:27.252511 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:27.253652 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:27.254060 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:27.254149 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:27.257961 (+ 3812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41263: BlockingRecv error: recv got EOF from 127.0.97.3:41263 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 28 08:38:27 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:27 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:27.333011 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:27.324895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.1:47083)
0928 08:38:27.325430 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:27.325446 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:27.325550 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:27.325844 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:27.325846 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:27.326247 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:27.326434 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:27.326442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:27.327444 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:27.327447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:27.328346 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:27.328357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:27.329117 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:27.329121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:27.329236 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:27.329906 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:27.329922 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:27.332637 (+ 2715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":319,"threads_started":1}
W20260928 08:38:27.333048 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:27.325013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47083 (local address 127.0.97.62:42985)
0928 08:38:27.325199 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:27.325203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:27.325518 (+ 315us) server_negotiation.cc:407] Connection header received
0928 08:38:27.325644 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:27.325648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:27.325718 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:27.325797 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:27.326616 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:27.327321 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:27.328513 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:27.328973 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:27.329073 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:27.332834 (+ 3761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47083: BlockingRecv error: recv got EOF from 127.0.97.1:47083 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 08:38:27 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:38:27 dist-test-slave-w6c0 krb5kdc[8642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:38:27.783653 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:27.775546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42985 (local address 127.0.97.2:45757)
0928 08:38:27.776017 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:27.776032 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:27.776184 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:27.776501 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:27.776506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:27.776927 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:27.777116 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:27.777124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:27.778146 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:27.778152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:27.779126 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:27.779137 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:27.779794 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:27.779800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:27.779968 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:27.780582 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:27.780601 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:27.783384 (+ 2783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42985: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":265,"threads_started":1}
W20260928 08:38:27.783828 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:27.775608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45757 (local address 127.0.97.62:42985)
0928 08:38:27.775817 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:27.775826 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:27.776121 (+ 295us) server_negotiation.cc:407] Connection header received
0928 08:38:27.776304 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:27.776311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:27.776374 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:27.776475 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:27.777248 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:27.777999 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:27.779267 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:27.779649 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:27.779786 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:27.783577 (+ 3791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45757: BlockingRecv error: recv got EOF from 127.0.97.2:45757 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 08:38:28.057941 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 9229
I20260928 08:38:28.069432 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:28.058890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44868 (local address 127.0.97.1:37691)
0928 08:38:28.059433 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:28.059440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:28.059465 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:38:28.059520 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:28.059528 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:28.059748 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:28.059896 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:28.061230 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:28.062215 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:28.063514 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:28.064044 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:28.064140 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:28.066656 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:28.066664 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:28.066677 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:28.066706 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:28.068957 (+ 2251us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 08:38:28.069213 (+ 256us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44868: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":287,"threads_started":1}
W20260928 08:38:28.069561 9689 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44868: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 08:38:28.069671 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 9229: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:37691: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 08:38:28.069885 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 9229
I20260928 08:38:28.086812 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 9380
I20260928 08:38:28.099327 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:28.087909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36160 (local address 127.0.97.2:41255)
0928 08:38:28.088512 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:28.088520 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:28.088551 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:38:28.088612 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:28.088619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:28.088861 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:28.089102 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:28.090407 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:28.091521 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:28.092751 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:28.093305 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:28.093450 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:28.096184 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:28.096193 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:28.096208 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:28.096245 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:28.098656 (+ 2411us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 08:38:28.098980 (+ 324us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36160: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":308,"threads_started":1}
W20260928 08:38:28.099336 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 9380: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:41255: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 08:38:28.099432 9690 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36160: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 08:38:28.099475 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 9380
I20260928 08:38:28.115929 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 9511
I20260928 08:38:28.127862 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:28.116900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55638 (local address 127.0.97.3:42027)
0928 08:38:28.117451 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:28.117458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:28.117483 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:38:28.117541 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:28.117547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:28.117771 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:28.117922 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:28.119269 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:28.120326 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:28.121761 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:28.122324 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:28.122433 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:28.124908 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:28.124916 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:28.124929 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:28.124979 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:28.127377 (+ 2398us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 08:38:28.127601 (+ 224us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55638: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":305,"threads_started":1}
W20260928 08:38:28.127969 9691 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55638: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 08:38:28.128031 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 9511: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:42027: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 08:38:28.128191 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 9511
I20260928 08:38:28.145134 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 9160
I20260928 08:38:28.146502 9222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:28.260891 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 9160
2026-09-28T08:38:28Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21169 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T08:38:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:38:28Z Disabled control of system clock
I20260928 08:38:28.336974 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36347
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 08:38:28.660893 9696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:28.661185 9696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:28.661334 9696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:28.668758 9696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:38:28.668869 9696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:28.668963 9696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:28.669057 9696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:38:28.669123 9696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:38:28.678484 9696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36347
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:28.679908 9696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:28.681699 9696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:28.694253 9702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:28.694327 9696 server_base.cc:1034] running on GCE node
W20260928 08:38:28.694350 9701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:28.694582 9704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:28.695451 9696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:28.696873 9696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:28.698112 9696 hybrid_clock.cc:648] HybridClock initialized: now 1790584708698083 us; error 51 us; skew 500 ppm
I20260928 08:38:28.698444 9696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:38:28.700659 9696 webserver.cc:460] Webserver started at http://127.0.97.62:44137/ using document root <none> and password file <none>
I20260928 08:38:28.701339 9696 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:28.701472 9696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:28.701792 9696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:28.704641 9696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "3703edf71292467484ed8d2487199f83"
format_stamp: "Formatted at 2026-09-28 08:38:28 on dist-test-slave-w6c0"
server_key: "af434f58ca5b791897a9bf274f4b6d32"
server_key_iv: "dc7e2213ade65546041b18d1fffd629f"
server_key_version: "encryptionkey@0"
I20260928 08:38:28.705417 9696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "3703edf71292467484ed8d2487199f83"
format_stamp: "Formatted at 2026-09-28 08:38:28 on dist-test-slave-w6c0"
server_key: "af434f58ca5b791897a9bf274f4b6d32"
server_key_iv: "dc7e2213ade65546041b18d1fffd629f"
server_key_version: "encryptionkey@0"
I20260928 08:38:28.710388 9696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:38:28.713796 9710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:28.715132 9696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:38:28.715318 9696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "3703edf71292467484ed8d2487199f83"
format_stamp: "Formatted at 2026-09-28 08:38:28 on dist-test-slave-w6c0"
server_key: "af434f58ca5b791897a9bf274f4b6d32"
server_key_iv: "dc7e2213ade65546041b18d1fffd629f"
server_key_version: "encryptionkey@0"
I20260928 08:38:28.715539 9696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:28.747460 9696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:28.748330 9696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:28.748629 9696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:28.766888 9696 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36347
I20260928 08:38:28.766913 9761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36347 every 8 connection(s)
I20260928 08:38:28.768165 9696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:38:28.771660 9762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:28.778175 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 9696
I20260928 08:38:28.778373 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:38:28.779224 388 external_mini_cluster.cc:1428] Setting key 85696572e0715332bd83950d65614718
I20260928 08:38:28.782445 9762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83: Bootstrap starting.
I20260928 08:38:28.785709 9762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:28.786757 9762 log.cc:824] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83: Log is configured to *not* fsync() on all Append() calls
I20260928 08:38:28.789714 9762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83: No bootstrap required, opened a new log
I20260928 08:38:28.792801 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:28.781255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44720 (local address 127.0.97.62:36347)
0928 08:38:28.782208 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:28.782219 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:28.782261 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:38:28.783938 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:28.783959 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:28.784380 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:28.784687 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:28.785351 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:28.786731 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:28.788367 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:28.788975 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:28.789101 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:28.790406 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:28.790413 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:28.790426 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:38:28.791005 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:28.791074 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:38:28.791100 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:28.791248 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:28.791487 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 08:38:28.791903 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":365,"threads_started":1}
I20260928 08:38:28.797621 9762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3703edf71292467484ed8d2487199f83" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36347 } }
I20260928 08:38:28.797943 9762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:28.798062 9762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3703edf71292467484ed8d2487199f83, State: Initialized, Role: FOLLOWER
I20260928 08:38:28.798686 9762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3703edf71292467484ed8d2487199f83" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36347 } }
I20260928 08:38:28.798904 9762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:38:28.799029 9762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:38:28.799199 9762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:28.801246 9762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3703edf71292467484ed8d2487199f83" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36347 } }
I20260928 08:38:28.801759 9762 leader_election.cc:302] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3703edf71292467484ed8d2487199f83; no voters:
I20260928 08:38:28.802428 9762 leader_election.cc:288] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:38:28.802525 9767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:38:28.803263 9767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83 [term 1 LEADER]: Becoming Leader. State: Replica: 3703edf71292467484ed8d2487199f83, State: Running, Role: LEADER
I20260928 08:38:28.803805 9767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3703edf71292467484ed8d2487199f83" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36347 } }
I20260928 08:38:28.804697 9762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:38:28.806541 9769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3703edf71292467484ed8d2487199f83. Latest consensus state: current_term: 1 leader_uuid: "3703edf71292467484ed8d2487199f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3703edf71292467484ed8d2487199f83" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36347 } } }
I20260928 08:38:28.806933 9769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83 [sys.catalog]: This master's current role is: LEADER
I20260928 08:38:28.807618 9768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3703edf71292467484ed8d2487199f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3703edf71292467484ed8d2487199f83" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36347 } } }
I20260928 08:38:28.808079 9768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83 [sys.catalog]: This master's current role is: LEADER
I20260928 08:38:28.809233 9775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:38:28.813936 9775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:38:28.824162 9775 catalog_manager.cc:1347] Generated new cluster ID: b891dc55e8bf415ebf9789f536a48f01
I20260928 08:38:28.824273 9775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:38:28.844818 9775 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:38:28.846616 9775 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:38:28.858640 9775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3703edf71292467484ed8d2487199f83: Generated new TSK 0
I20260928 08:38:28.859666 9775 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:38:28.866052 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36347
--builtin_ntp_servers=127.0.97.20:43159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 08:38:29.192525 9786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:29.192795 9786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:29.192879 9786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:29.201352 9786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:29.201502 9786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:29.201714 9786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:38:29.212528 9786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:29.213798 9786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:29.215725 9786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:29.228938 9794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:29.228971 9792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:29.229024 9791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:29.231923 9786 server_base.cc:1034] running on GCE node
I20260928 08:38:29.232558 9786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:29.233461 9786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:29.234645 9786 hybrid_clock.cc:648] HybridClock initialized: now 1790584709234595 us; error 37 us; skew 500 ppm
I20260928 08:38:29.234928 9786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:38:29.244112 9786 webserver.cc:460] Webserver started at http://127.0.97.1:32873/ using document root <none> and password file <none>
I20260928 08:38:29.244764 9786 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:29.244855 9786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:29.245119 9786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:29.247877 9786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "a2d3d13933954391944dd270d277b861"
format_stamp: "Formatted at 2026-09-28 08:38:29 on dist-test-slave-w6c0"
server_key: "99029ec778ed38276220d3dc729735bb"
server_key_iv: "326392e75b2f3f1da80bf0a2491b1949"
server_key_version: "encryptionkey@0"
I20260928 08:38:29.248605 9786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a2d3d13933954391944dd270d277b861"
format_stamp: "Formatted at 2026-09-28 08:38:29 on dist-test-slave-w6c0"
server_key: "99029ec778ed38276220d3dc729735bb"
server_key_iv: "326392e75b2f3f1da80bf0a2491b1949"
server_key_version: "encryptionkey@0"
I20260928 08:38:29.253402 9786 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 08:38:29.256925 9800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:29.258493 9786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:38:29.258735 9786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "a2d3d13933954391944dd270d277b861"
format_stamp: "Formatted at 2026-09-28 08:38:29 on dist-test-slave-w6c0"
server_key: "99029ec778ed38276220d3dc729735bb"
server_key_iv: "326392e75b2f3f1da80bf0a2491b1949"
server_key_version: "encryptionkey@0"
I20260928 08:38:29.258910 9786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:29.271647 9786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:29.272401 9786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:29.272620 9786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:29.273556 9786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:38:29.275050 9786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:38:29.275135 9786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:29.275216 9786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:38:29.275269 9786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:29.308759 9786 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33919
I20260928 08:38:29.308830 9912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33919 every 8 connection(s)
I20260928 08:38:29.310025 9786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:38:29.317646 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 9786
I20260928 08:38:29.317819 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:38:29.318650 388 external_mini_cluster.cc:1428] Setting key b328b4ed52c7120d480af9f658bd1f91
I20260928 08:38:29.321543 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36347
--builtin_ntp_servers=127.0.97.20:43159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 08:38:29.323289 9915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:29.312526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39795 (local address 127.0.97.62:36347)
0928 08:38:29.312942 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:29.312954 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:29.314169 (+ 1215us) server_negotiation.cc:407] Connection header received
0928 08:38:29.316018 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:29.316022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:29.316091 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:29.316188 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:29.317154 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:29.318019 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:29.320334 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:29.320788 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:29.320889 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:29.322462 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:29.322467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:29.322470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:38:29.322486 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:29.322604 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:38:29.322618 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:29.322713 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:29.322977 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 08:38:29.323048 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":217,"threads_started":1}
I20260928 08:38:29.324651 9916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:29.313108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36347 (local address 127.0.97.1:39795)
0928 08:38:29.314009 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:29.314047 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:29.315768 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:29.316370 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:29.316377 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:29.316456 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:29.317001 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:29.317018 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:29.318173 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:29.318179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:29.320119 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:29.320132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:29.321011 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:29.321018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:29.321328 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:29.322086 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:38:29.322092 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:29.322143 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:38:29.322244 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:38:29.322734 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:38:29.322748 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:38:29.322760 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 08:38:29.322953 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 08:38:29.323349 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":265,"threads_started":1}
I20260928 08:38:29.326103 9913 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36347
I20260928 08:38:29.326398 9913 heartbeater.cc:464] Registering TS with master...
I20260928 08:38:29.327265 9913 heartbeater.cc:511] Master 127.0.97.62:36347 requested a full tablet report, sending...
I20260928 08:38:29.329535 9727 ts_manager.cc:194] Registered new tserver with Master: a2d3d13933954391944dd270d277b861 (127.0.97.1:33919)
I20260928 08:38:29.331853 9727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:39795
W20260928 08:38:29.644445 9917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:29.644714 9917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:29.644795 9917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:29.651952 9917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:29.652052 9917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:29.652155 9917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:38:29.661203 9917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:29.662542 9917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:29.664290 9917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:29.677157 9923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:29.677179 9922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:29.677778 9917 server_base.cc:1034] running on GCE node
W20260928 08:38:29.678038 9925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:29.678776 9917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:29.679726 9917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:29.680943 9917 hybrid_clock.cc:648] HybridClock initialized: now 1790584709680903 us; error 61 us; skew 500 ppm
I20260928 08:38:29.681252 9917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:38:29.683473 9917 webserver.cc:460] Webserver started at http://127.0.97.2:41655/ using document root <none> and password file <none>
I20260928 08:38:29.684239 9917 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:29.684368 9917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:29.684688 9917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:29.687496 9917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "bff359cfb0db4ec3bd990999a7ee436c"
format_stamp: "Formatted at 2026-09-28 08:38:29 on dist-test-slave-w6c0"
server_key: "d5137dab8ddc7c79f5e489f65ee82b63"
server_key_iv: "f93870350a12bb38de80c8965c1eaaca"
server_key_version: "encryptionkey@0"
I20260928 08:38:29.688239 9917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bff359cfb0db4ec3bd990999a7ee436c"
format_stamp: "Formatted at 2026-09-28 08:38:29 on dist-test-slave-w6c0"
server_key: "d5137dab8ddc7c79f5e489f65ee82b63"
server_key_iv: "f93870350a12bb38de80c8965c1eaaca"
server_key_version: "encryptionkey@0"
I20260928 08:38:29.693078 9917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 08:38:29.696429 9931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:29.697649 9917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:38:29.697815 9917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "bff359cfb0db4ec3bd990999a7ee436c"
format_stamp: "Formatted at 2026-09-28 08:38:29 on dist-test-slave-w6c0"
server_key: "d5137dab8ddc7c79f5e489f65ee82b63"
server_key_iv: "f93870350a12bb38de80c8965c1eaaca"
server_key_version: "encryptionkey@0"
I20260928 08:38:29.698016 9917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:29.724061 9917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:29.724923 9917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:29.725199 9917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:29.726135 9917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:38:29.727674 9917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:38:29.727793 9917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:29.727932 9917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:38:29.728020 9917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:29.760007 9917 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37301
I20260928 08:38:29.760090 10043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37301 every 8 connection(s)
I20260928 08:38:29.761303 9917 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:38:29.765661 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 9917
I20260928 08:38:29.765838 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:38:29.766387 388 external_mini_cluster.cc:1428] Setting key ff395781a7f65653dfcea3dc74c20149
I20260928 08:38:29.769539 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36347
--builtin_ntp_servers=127.0.97.20:43159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 08:38:29.774299 9915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:29.763933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46673 (local address 127.0.97.62:36347)
0928 08:38:29.764149 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:29.764155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:29.765576 (+ 1421us) server_negotiation.cc:407] Connection header received
0928 08:38:29.768043 (+ 2467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:29.768048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:29.768124 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:29.768235 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:29.769314 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:29.770141 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:29.771710 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:29.772111 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:29.772250 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:29.773505 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:29.773509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:29.773512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:38:29.773529 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:29.773555 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:38:29.773568 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:29.773658 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:29.773888 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 08:38:29.773990 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:38:29.775403 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:29.764464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36347 (local address 127.0.97.2:46673)
0928 08:38:29.765395 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:29.765454 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:29.767749 (+ 2295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:29.768493 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:29.768505 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:29.768623 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:29.769151 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:29.769167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:29.770294 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:29.770301 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:29.771571 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:29.771582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:29.772323 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:29.772330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:29.772544 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:29.773209 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:38:29.773211 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:29.773250 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:38:29.773321 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:38:29.773657 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:38:29.773667 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:38:29.773677 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 08:38:29.773886 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 08:38:29.774247 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":263,"threads_started":1}
I20260928 08:38:29.776710 10044 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36347
I20260928 08:38:29.777016 10044 heartbeater.cc:464] Registering TS with master...
I20260928 08:38:29.777717 10044 heartbeater.cc:511] Master 127.0.97.62:36347 requested a full tablet report, sending...
I20260928 08:38:29.779393 9727 ts_manager.cc:194] Registered new tserver with Master: bff359cfb0db4ec3bd990999a7ee436c (127.0.97.2:37301)
I20260928 08:38:29.780470 9727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:46673
W20260928 08:38:30.087445 10047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:30.087755 10047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:30.087914 10047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:30.095099 10047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:30.095225 10047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:30.095404 10047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:38:30.104423 10047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:30.105760 10047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:30.107601 10047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:30.120460 10053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:30.120527 10055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:30.120754 10052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:30.121167 10047 server_base.cc:1034] running on GCE node
I20260928 08:38:30.121764 10047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:30.122622 10047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:30.123811 10047 hybrid_clock.cc:648] HybridClock initialized: now 1790584710123757 us; error 49 us; skew 500 ppm
I20260928 08:38:30.124106 10047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:38:30.126230 10047 webserver.cc:460] Webserver started at http://127.0.97.3:42081/ using document root <none> and password file <none>
I20260928 08:38:30.126940 10047 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:30.127059 10047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:30.127377 10047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:30.130014 10047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "b7b014c701c3431ebc8649589656bd7e"
format_stamp: "Formatted at 2026-09-28 08:38:30 on dist-test-slave-w6c0"
server_key: "da6ac3a0530a8cb84f49c5d735599ed5"
server_key_iv: "0a049edf151ac3f908d8924888b504d1"
server_key_version: "encryptionkey@0"
I20260928 08:38:30.130833 10047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b7b014c701c3431ebc8649589656bd7e"
format_stamp: "Formatted at 2026-09-28 08:38:30 on dist-test-slave-w6c0"
server_key: "da6ac3a0530a8cb84f49c5d735599ed5"
server_key_iv: "0a049edf151ac3f908d8924888b504d1"
server_key_version: "encryptionkey@0"
I20260928 08:38:30.135443 10047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 08:38:30.138687 10061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:30.139824 10047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 08:38:30.139983 10047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "b7b014c701c3431ebc8649589656bd7e"
format_stamp: "Formatted at 2026-09-28 08:38:30 on dist-test-slave-w6c0"
server_key: "da6ac3a0530a8cb84f49c5d735599ed5"
server_key_iv: "0a049edf151ac3f908d8924888b504d1"
server_key_version: "encryptionkey@0"
I20260928 08:38:30.140216 10047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:30.183259 10047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:30.184131 10047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:30.184424 10047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:30.185377 10047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:38:30.186822 10047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:38:30.186945 10047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:30.187085 10047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:38:30.187178 10047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:30.219580 10047 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44395
I20260928 08:38:30.219668 10173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44395 every 8 connection(s)
I20260928 08:38:30.220834 10047 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:38:30.224828 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 10047
I20260928 08:38:30.224999 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:38:30.225540 388 external_mini_cluster.cc:1428] Setting key f040e98a7920a6926563effd1f73b4ff
I20260928 08:38:30.233090 9915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:30.223269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35759 (local address 127.0.97.62:36347)
0928 08:38:30.223477 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:30.223483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:30.225266 (+ 1783us) server_negotiation.cc:407] Connection header received
0928 08:38:30.227160 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:30.227164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:30.227284 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:30.227380 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:30.228579 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:30.229302 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:30.230807 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:30.231195 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:30.231284 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:30.232415 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:30.232420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:30.232423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:38:30.232438 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:30.232467 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:38:30.232478 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:30.232543 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:30.232755 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 08:38:30.232825 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:38:30.233922 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:30.223838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36347 (local address 127.0.97.3:35759)
0928 08:38:30.224801 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:30.224857 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:30.226878 (+ 2021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:30.227635 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:30.227644 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:30.227758 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:30.228370 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:30.228413 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:30.229446 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:30.229449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:30.230685 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:30.230696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:30.231360 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:30.231364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:30.231542 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:30.232160 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:38:30.232163 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:30.232201 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:38:30.232266 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:38:30.232557 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:38:30.232563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:38:30.232569 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 08:38:30.232703 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 08:38:30.232958 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":260,"threads_started":1}
I20260928 08:38:30.235113 10174 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36347
I20260928 08:38:30.235395 10174 heartbeater.cc:464] Registering TS with master...
I20260928 08:38:30.236097 10174 heartbeater.cc:511] Master 127.0.97.62:36347 requested a full tablet report, sending...
I20260928 08:38:30.237439 9727 ts_manager.cc:194] Registered new tserver with Master: b7b014c701c3431ebc8649589656bd7e (127.0.97.3:44395)
I20260928 08:38:30.238217 9727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:35759
I20260928 08:38:30.240522 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 08:38:30.246411 9915 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:30.245012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44722 (local address 127.0.97.62:36347)
0928 08:38:30.245198 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:30.245207 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:30.245608 (+ 401us) server_negotiation.cc:407] Connection header received
0928 08:38:30.245707 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:30.245710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:30.245769 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:30.245846 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:30.246197 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44722: BlockingRecv error: recv got EOF from 127.0.0.1:44722 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:38:30.247080 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 9786
I20260928 08:38:30.253811 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:30.247909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.0.97.1:33919)
0928 08:38:30.248377 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:30.248383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:30.248406 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:38:30.248455 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:30.248460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:30.248624 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:30.248742 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:30.249037 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:30.249952 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:30.251139 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:30.251654 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:30.251760 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:30.252635 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:30.252642 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:30.252653 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:38:30.253177 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:30.253215 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:38:30.253231 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:30.253317 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:30.253448 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 08:38:30.253536 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":233,"threads_started":1}
I20260928 08:38:30.254954 9908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:30.334673 9913 heartbeater.cc:503] Master 127.0.97.62:36347 was elected leader, sending a full tablet report...
I20260928 08:38:30.336239 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 9786
I20260928 08:38:30.352481 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 9917
I20260928 08:38:30.360181 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:30.353332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54906 (local address 127.0.97.2:37301)
0928 08:38:30.353836 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:30.353843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:30.353867 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:38:30.353921 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:30.353927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:30.354140 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:30.354318 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:30.354765 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:30.355751 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:30.356938 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:30.357491 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:30.357623 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:30.358776 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:30.358791 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:30.358811 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:38:30.359357 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:30.359403 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:38:30.359421 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:30.359559 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:30.359692 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 08:38:30.359820 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":264,"threads_started":1}
I20260928 08:38:30.361378 10039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:30.443976 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 9917
I20260928 08:38:30.459959 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 10047
I20260928 08:38:30.467613 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:30.460846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42360 (local address 127.0.97.3:44395)
0928 08:38:30.461454 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:30.461461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:30.461490 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:38:30.461559 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:30.461565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:30.461781 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:30.461932 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:30.462341 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:30.463428 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:30.464595 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:30.465110 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:30.465202 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:30.466159 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:30.466165 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:30.466177 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:38:30.466843 (+ 666us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:30.466888 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:38:30.466906 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:30.466997 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:30.467123 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 08:38:30.467255 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":322,"threads_started":1}
I20260928 08:38:30.468901 10169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:30.550417 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 10047
I20260928 08:38:30.566071 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 9696
I20260928 08:38:30.567255 9757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:30.668495 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 9696
2026-09-28T08:38:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2401 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T08:38:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:38:30Z Disabled control of system clock
I20260928 08:38:30.736402 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40009
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 08:38:31.043212 10193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:31.043488 10193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:31.043572 10193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:31.050580 10193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:38:31.050736 10193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:31.050786 10193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:31.050839 10193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:38:31.050889 10193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:38:31.059623 10193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40009
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:31.060883 10193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:31.062609 10193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:31.073920 10198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:31.073982 10201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:31.073956 10199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:31.074764 10193 server_base.cc:1034] running on GCE node
I20260928 08:38:31.075485 10193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:31.076803 10193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:31.077999 10193 hybrid_clock.cc:648] HybridClock initialized: now 1790584711077964 us; error 50 us; skew 500 ppm
I20260928 08:38:31.078330 10193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:38:31.080469 10193 webserver.cc:460] Webserver started at http://127.0.97.62:46615/ using document root <none> and password file <none>
I20260928 08:38:31.081135 10193 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:31.081246 10193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:31.081564 10193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:31.084439 10193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "4c728257e40b400ebb08593e9077f2ce"
format_stamp: "Formatted at 2026-09-28 08:38:31 on dist-test-slave-w6c0"
server_key: "7e5ce9331dc2495ac831fc3ce18b48fe"
server_key_iv: "7a376d5a63a529089944f99f7a302f0d"
server_key_version: "encryptionkey@0"
I20260928 08:38:31.085212 10193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "4c728257e40b400ebb08593e9077f2ce"
format_stamp: "Formatted at 2026-09-28 08:38:31 on dist-test-slave-w6c0"
server_key: "7e5ce9331dc2495ac831fc3ce18b48fe"
server_key_iv: "7a376d5a63a529089944f99f7a302f0d"
server_key_version: "encryptionkey@0"
I20260928 08:38:31.090039 10193 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260928 08:38:31.093163 10207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:31.094310 10193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 08:38:31.094487 10193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "4c728257e40b400ebb08593e9077f2ce"
format_stamp: "Formatted at 2026-09-28 08:38:31 on dist-test-slave-w6c0"
server_key: "7e5ce9331dc2495ac831fc3ce18b48fe"
server_key_iv: "7a376d5a63a529089944f99f7a302f0d"
server_key_version: "encryptionkey@0"
I20260928 08:38:31.094777 10193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:31.100159 10193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:31.100863 10193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:31.101147 10193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:31.118953 10193 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40009
I20260928 08:38:31.118986 10258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40009 every 8 connection(s)
I20260928 08:38:31.120256 10193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:38:31.123806 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 10193
I20260928 08:38:31.123837 10259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:31.124018 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:38:31.124637 388 external_mini_cluster.cc:1428] Setting key 5476c31937e86370e21bd616cba162d4
I20260928 08:38:31.132630 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:31.126496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.0.97.62:40009)
0928 08:38:31.127561 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:31.127570 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:31.127611 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:38:31.129436 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:31.129455 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:31.129857 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:31.130153 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:31.131181 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:31.131187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:31.131198 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:38:31.131211 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:31.131267 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:38:31.131281 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:31.131433 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:31.131552 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 08:38:31.131837 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":395,"threads_started":1}
I20260928 08:38:31.136327 10259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce: Bootstrap starting.
I20260928 08:38:31.139150 10259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:31.140234 10259 log.cc:824] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce: Log is configured to *not* fsync() on all Append() calls
I20260928 08:38:31.142848 10259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce: No bootstrap required, opened a new log
I20260928 08:38:31.149850 10259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c728257e40b400ebb08593e9077f2ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40009 } }
I20260928 08:38:31.150213 10259 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:31.150313 10259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c728257e40b400ebb08593e9077f2ce, State: Initialized, Role: FOLLOWER
I20260928 08:38:31.150887 10259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c728257e40b400ebb08593e9077f2ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40009 } }
I20260928 08:38:31.151134 10259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:38:31.151271 10259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:38:31.151449 10259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:31.153725 10259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c728257e40b400ebb08593e9077f2ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40009 } }
I20260928 08:38:31.154228 10259 leader_election.cc:302] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c728257e40b400ebb08593e9077f2ce; no voters:
I20260928 08:38:31.154879 10259 leader_election.cc:288] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:38:31.154973 10264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:38:31.155759 10264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce [term 1 LEADER]: Becoming Leader. State: Replica: 4c728257e40b400ebb08593e9077f2ce, State: Running, Role: LEADER
I20260928 08:38:31.156332 10264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c728257e40b400ebb08593e9077f2ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40009 } }
I20260928 08:38:31.157047 10259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:38:31.159170 10266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c728257e40b400ebb08593e9077f2ce. Latest consensus state: current_term: 1 leader_uuid: "4c728257e40b400ebb08593e9077f2ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c728257e40b400ebb08593e9077f2ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40009 } } }
I20260928 08:38:31.159276 10265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c728257e40b400ebb08593e9077f2ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c728257e40b400ebb08593e9077f2ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40009 } } }
I20260928 08:38:31.159569 10266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce [sys.catalog]: This master's current role is: LEADER
I20260928 08:38:31.159566 10265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce [sys.catalog]: This master's current role is: LEADER
I20260928 08:38:31.163858 10272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:38:31.168035 10272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:38:31.176600 10272 catalog_manager.cc:1347] Generated new cluster ID: aa7087cfcf8144f2b5b76ba6d7f80cb8
I20260928 08:38:31.176718 10272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:38:31.209120 10272 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:38:31.209663 10272 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:38:31.219040 10272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c728257e40b400ebb08593e9077f2ce: Generated new TSK 0
I20260928 08:38:31.220011 10272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:38:31.232805 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40009
--builtin_ntp_servers=127.0.97.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 08:38:31.556612 10283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:31.556877 10283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:31.557049 10283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:31.564713 10283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:31.564803 10283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:31.565007 10283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:38:31.573966 10283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:31.575248 10283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:31.576984 10283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:31.589862 10289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:31.590211 10291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:31.590250 10288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:31.590777 10283 server_base.cc:1034] running on GCE node
I20260928 08:38:31.591390 10283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:31.592242 10283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:31.593483 10283 hybrid_clock.cc:648] HybridClock initialized: now 1790584711593425 us; error 70 us; skew 500 ppm
I20260928 08:38:31.593825 10283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:38:31.596125 10283 webserver.cc:460] Webserver started at http://127.0.97.1:46217/ using document root <none> and password file <none>
I20260928 08:38:31.596802 10283 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:31.596925 10283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:31.597239 10283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:31.601595 10283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "b8185fa1962b4d4e9c546198d87f0153"
format_stamp: "Formatted at 2026-09-28 08:38:31 on dist-test-slave-w6c0"
server_key: "0aa8d63b80992be819f0d5d9a56fec79"
server_key_iv: "14ed58129158153126b73466a6a39945"
server_key_version: "encryptionkey@0"
I20260928 08:38:31.602438 10283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b8185fa1962b4d4e9c546198d87f0153"
format_stamp: "Formatted at 2026-09-28 08:38:31 on dist-test-slave-w6c0"
server_key: "0aa8d63b80992be819f0d5d9a56fec79"
server_key_iv: "14ed58129158153126b73466a6a39945"
server_key_version: "encryptionkey@0"
I20260928 08:38:31.607259 10283 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 08:38:31.610452 10297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:31.611732 10283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:38:31.611896 10283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "b8185fa1962b4d4e9c546198d87f0153"
format_stamp: "Formatted at 2026-09-28 08:38:31 on dist-test-slave-w6c0"
server_key: "0aa8d63b80992be819f0d5d9a56fec79"
server_key_iv: "14ed58129158153126b73466a6a39945"
server_key_version: "encryptionkey@0"
I20260928 08:38:31.612111 10283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:31.617538 10283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:31.618265 10283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:31.618571 10283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:31.619418 10283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:38:31.620707 10283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:38:31.620834 10283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:31.620954 10283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:38:31.621047 10283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:31.652689 10283 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43365
I20260928 08:38:31.652740 10409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43365 every 8 connection(s)
I20260928 08:38:31.653991 10283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:38:31.655637 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 10283
I20260928 08:38:31.655839 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:38:31.656558 388 external_mini_cluster.cc:1428] Setting key 2082fc11aab301c233dafff38f45c653
I20260928 08:38:31.659842 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40009
--builtin_ntp_servers=127.0.97.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 08:38:31.662603 10412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:31.656613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39515 (local address 127.0.97.62:40009)
0928 08:38:31.658884 (+ 2271us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:31.658890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:31.658917 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:38:31.660162 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:31.660167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:31.660246 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:31.660360 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:31.661841 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:31.661846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:31.661857 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:38:31.661872 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:31.661910 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:38:31.661921 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:31.662040 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:31.662237 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 08:38:31.662301 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2155,"thread_start_us":2054,"threads_started":1}
I20260928 08:38:31.663573 10413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:31.657028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40009 (local address 127.0.97.1:39515)
0928 08:38:31.658092 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:31.658140 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:31.659846 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:31.660644 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:31.660656 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:31.660793 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:31.661564 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:38:31.661567 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:31.661601 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:38:31.661677 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:38:31.662014 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:38:31.662025 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:38:31.662032 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 08:38:31.662328 (+ 296us) client_negotiation.cc:240] Negotiation successful
0928 08:38:31.662635 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":371,"threads_started":1}
I20260928 08:38:31.665150 10410 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40009
I20260928 08:38:31.665429 10410 heartbeater.cc:464] Registering TS with master...
I20260928 08:38:31.665975 10410 heartbeater.cc:511] Master 127.0.97.62:40009 requested a full tablet report, sending...
I20260928 08:38:31.667905 10224 ts_manager.cc:194] Registered new tserver with Master: b8185fa1962b4d4e9c546198d87f0153 (127.0.97.1:43365)
W20260928 08:38:31.975775 10414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:31.976100 10414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:31.976248 10414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:31.983577 10414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:31.983702 10414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:31.983889 10414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:38:31.992903 10414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:31.994158 10414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:31.995963 10414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:32.008970 10422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:32.009032 10419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:32.009028 10420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:32.010131 10414 server_base.cc:1034] running on GCE node
I20260928 08:38:32.010751 10414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:32.011747 10414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:32.012965 10414 hybrid_clock.cc:648] HybridClock initialized: now 1790584712012905 us; error 59 us; skew 500 ppm
I20260928 08:38:32.013239 10414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:38:32.015416 10414 webserver.cc:460] Webserver started at http://127.0.97.2:44585/ using document root <none> and password file <none>
I20260928 08:38:32.016054 10414 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:32.016131 10414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:32.016430 10414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:32.019173 10414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "3d4af6214e5d4aa19000d3ec7c8c0594"
format_stamp: "Formatted at 2026-09-28 08:38:32 on dist-test-slave-w6c0"
server_key: "41ee855554b09992fa5ddc68450b4dde"
server_key_iv: "d97b440383edf7ba4eefe72d64f8fbd9"
server_key_version: "encryptionkey@0"
I20260928 08:38:32.019881 10414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3d4af6214e5d4aa19000d3ec7c8c0594"
format_stamp: "Formatted at 2026-09-28 08:38:32 on dist-test-slave-w6c0"
server_key: "41ee855554b09992fa5ddc68450b4dde"
server_key_iv: "d97b440383edf7ba4eefe72d64f8fbd9"
server_key_version: "encryptionkey@0"
I20260928 08:38:32.024663 10414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 08:38:32.027797 10428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:32.029052 10414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 08:38:32.029176 10414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "3d4af6214e5d4aa19000d3ec7c8c0594"
format_stamp: "Formatted at 2026-09-28 08:38:32 on dist-test-slave-w6c0"
server_key: "41ee855554b09992fa5ddc68450b4dde"
server_key_iv: "d97b440383edf7ba4eefe72d64f8fbd9"
server_key_version: "encryptionkey@0"
I20260928 08:38:32.029352 10414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:32.036587 10414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:32.037575 10414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:32.037875 10414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:32.039044 10414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:38:32.040814 10414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:38:32.040931 10414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:32.041043 10414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:38:32.041137 10414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:32.073446 10414 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43829
I20260928 08:38:32.073496 10540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43829 every 8 connection(s)
I20260928 08:38:32.074852 10414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:38:32.081465 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 10414
I20260928 08:38:32.081642 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:38:32.082170 388 external_mini_cluster.cc:1428] Setting key 6bc4af7f7e9ab3b8d077f6426f2167f4
I20260928 08:38:32.082476 10412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:32.077268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48895 (local address 127.0.97.62:40009)
0928 08:38:32.077476 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:32.077482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:32.078857 (+ 1375us) server_negotiation.cc:407] Connection header received
0928 08:38:32.080409 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:32.080414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:32.080463 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:32.080549 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:32.081812 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:32.081816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:32.081819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:38:32.081834 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:32.081864 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:38:32.081874 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:32.081962 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:32.082172 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 08:38:32.082219 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:38:32.083366 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:32.077775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40009 (local address 127.0.97.2:48895)
0928 08:38:32.078713 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:32.078753 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:32.080191 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:32.080731 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:32.080738 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:32.080817 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:32.081464 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:38:32.081467 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:32.081512 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:38:32.081651 (+ 139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:38:32.081971 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:38:32.081981 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:38:32.081989 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 08:38:32.082251 (+ 262us) client_negotiation.cc:240] Negotiation successful
0928 08:38:32.082480 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":302,"threads_started":1}
I20260928 08:38:32.084640 10541 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40009
I20260928 08:38:32.084918 10541 heartbeater.cc:464] Registering TS with master...
I20260928 08:38:32.084909 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40009
--builtin_ntp_servers=127.0.97.20:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 08:38:32.085484 10541 heartbeater.cc:511] Master 127.0.97.62:40009 requested a full tablet report, sending...
I20260928 08:38:32.086813 10224 ts_manager.cc:194] Registered new tserver with Master: 3d4af6214e5d4aa19000d3ec7c8c0594 (127.0.97.2:43829)
W20260928 08:38:32.405270 10544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:32.405570 10544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:32.405736 10544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:32.412914 10544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:32.413000 10544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:32.413112 10544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:38:32.421994 10544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:32.423223 10544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:32.424854 10544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:32.437358 10549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:32.437304 10550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:32.437390 10552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:32.438084 10544 server_base.cc:1034] running on GCE node
I20260928 08:38:32.438725 10544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:32.439606 10544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:32.440807 10544 hybrid_clock.cc:648] HybridClock initialized: now 1790584712440765 us; error 42 us; skew 500 ppm
I20260928 08:38:32.441154 10544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:38:32.443418 10544 webserver.cc:460] Webserver started at http://127.0.97.3:46061/ using document root <none> and password file <none>
I20260928 08:38:32.444084 10544 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:32.444195 10544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:32.444502 10544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:32.447332 10544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "1167be786c7f4d43b889f5258734768e"
format_stamp: "Formatted at 2026-09-28 08:38:32 on dist-test-slave-w6c0"
server_key: "fb782b2c257305b075b1985525de476f"
server_key_iv: "a3af071a1edde1926a5c7e3c4a2cf947"
server_key_version: "encryptionkey@0"
I20260928 08:38:32.448082 10544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1167be786c7f4d43b889f5258734768e"
format_stamp: "Formatted at 2026-09-28 08:38:32 on dist-test-slave-w6c0"
server_key: "fb782b2c257305b075b1985525de476f"
server_key_iv: "a3af071a1edde1926a5c7e3c4a2cf947"
server_key_version: "encryptionkey@0"
I20260928 08:38:32.452950 10544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 08:38:32.456228 10558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:32.457535 10544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:38:32.457710 10544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "1167be786c7f4d43b889f5258734768e"
format_stamp: "Formatted at 2026-09-28 08:38:32 on dist-test-slave-w6c0"
server_key: "fb782b2c257305b075b1985525de476f"
server_key_iv: "a3af071a1edde1926a5c7e3c4a2cf947"
server_key_version: "encryptionkey@0"
I20260928 08:38:32.457940 10544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:32.463605 10544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:32.464407 10544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:32.464670 10544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:32.465535 10544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:38:32.466898 10544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:38:32.467015 10544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:32.467159 10544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:38:32.467248 10544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:32.499655 10544 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34267
I20260928 08:38:32.499738 10670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34267 every 8 connection(s)
I20260928 08:38:32.500959 10544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:38:32.505869 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 10544
I20260928 08:38:32.506062 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:38:32.506469 388 external_mini_cluster.cc:1428] Setting key d15201060f592f9a5f9bb27f0ff46d45
I20260928 08:38:32.509191 10412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:32.503507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53035 (local address 127.0.97.62:40009)
0928 08:38:32.503695 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:32.503700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:32.505000 (+ 1300us) server_negotiation.cc:407] Connection header received
0928 08:38:32.506832 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:32.506839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:32.506906 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:32.507022 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:32.508551 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:32.508556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:32.508562 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:38:32.508584 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:32.508613 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:38:32.508624 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:32.508716 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:32.508932 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 08:38:32.508979 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:38:32.510205 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:32.504005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40009 (local address 127.0.97.3:53035)
0928 08:38:32.504854 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:32.504888 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:32.506479 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:32.507292 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:32.507300 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:32.507386 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:32.508210 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:38:32.508213 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:32.508263 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:38:32.508390 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:38:32.508726 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:38:32.508736 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:38:32.508743 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 08:38:32.509015 (+ 272us) client_negotiation.cc:240] Negotiation successful
0928 08:38:32.509270 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":258,"threads_started":1}
I20260928 08:38:32.511394 10671 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40009
I20260928 08:38:32.511641 10671 heartbeater.cc:464] Registering TS with master...
I20260928 08:38:32.512149 10671 heartbeater.cc:511] Master 127.0.97.62:40009 requested a full tablet report, sending...
I20260928 08:38:32.513358 10224 ts_manager.cc:194] Registered new tserver with Master: 1167be786c7f4d43b889f5258734768e (127.0.97.3:34267)
I20260928 08:38:32.521533 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 08:38:32.527321 10412 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:38:32.525868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.0.97.62:40009)
0928 08:38:32.526018 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:32.526022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:32.526476 (+ 454us) server_negotiation.cc:407] Connection header received
0928 08:38:32.526672 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:32.526676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:32.526727 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:32.526804 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:32.527051 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60506: BlockingRecv error: recv got EOF from 127.0.0.1:60506 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 08:38:32.528036 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 10283
I20260928 08:38:32.531353 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:32.528975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.0.97.1:43365)
0928 08:38:32.529427 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:32.529434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:32.529457 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:38:32.529510 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:32.529515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:32.529708 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:32.529822 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:32.530707 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:32.530713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:32.530723 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:38:32.530736 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:32.530782 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:38:32.530799 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:32.530897 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:32.530972 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 08:38:32.531077 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":226,"threads_started":1}
I20260928 08:38:32.532445 10405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:32.612948 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 10283
I20260928 08:38:32.629426 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 10414
I20260928 08:38:32.633013 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:32.630477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35918 (local address 127.0.97.2:43829)
0928 08:38:32.630975 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:32.630983 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:32.631017 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:38:32.631076 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:32.631082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:32.631298 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:32.631424 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:32.632324 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:32.632330 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:32.632340 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:38:32.632359 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:32.632405 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:38:32.632419 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:32.632523 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:32.632608 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 08:38:32.632729 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":236,"threads_started":1}
I20260928 08:38:32.634228 10536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:32.715760 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 10414
I20260928 08:38:32.732122 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 10544
I20260928 08:38:32.736135 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:32.733158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55132 (local address 127.0.97.3:34267)
0928 08:38:32.733747 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:32.733754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:32.733783 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:38:32.733851 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:32.733857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:32.734075 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:32.734254 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:32.735402 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:32.735408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:32.735419 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:38:32.735434 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:32.735470 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:38:32.735483 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:32.735677 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:32.735748 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 08:38:32.735845 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":291,"threads_started":1}
I20260928 08:38:32.737329 10666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:32.819347 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 10544
I20260928 08:38:32.834991 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 10193
I20260928 08:38:32.836304 10254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:32.936717 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 10193
2026-09-28T08:38:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2267 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:38:33 dist-test-slave-w6c0 krb5kdc[10689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:38:33 dist-test-slave-w6c0 krb5kdc[10689](info): set up 2 sockets
Sep 28 08:38:33 dist-test-slave-w6c0 krb5kdc[10689](info): commencing operation
krb5kdc: starting...
W20260928 08:38:34.055599 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:38:34 dist-test-slave-w6c0 krb5kdc[10689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:38:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:38:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:38:34.304596 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35051
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:38:34.638837 10702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:34.639228 10702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:34.639402 10702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:34.647315 10702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:38:34.647473 10702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:34.647559 10702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:34.647641 10702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:38:34.647710 10702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:38:34.657673 10702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:34.659157 10702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:34.660522 10702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:34.672552 10707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:34.672605 10708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:34.672581 10710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:34.673420 10702 server_base.cc:1034] running on GCE node
I20260928 08:38:34.674228 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:34.675850 10702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:34.677088 10702 hybrid_clock.cc:648] HybridClock initialized: now 1790584714677035 us; error 52 us; skew 500 ppm
I20260928 08:38:34.677445 10702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:38:34 dist-test-slave-w6c0 krb5kdc[10689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:38:34.686142 10702 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:38:34.687803 10702 webserver.cc:460] Webserver started at http://127.0.97.62:34093/ using document root <none> and password file <none>
I20260928 08:38:34.688515 10702 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:34.688664 10702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:34.688993 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:34.692014 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "b7be6582fb6648e49231ea56fe1d0e2e"
format_stamp: "Formatted at 2026-09-28 08:38:34 on dist-test-slave-w6c0"
server_key: "c49fd610f6e1af0afbcac3317aa57ec6"
server_key_iv: "8287ea0573ddaaf3584e5f74a0733da4"
server_key_version: "encryptionkey@0"
I20260928 08:38:34.692790 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "b7be6582fb6648e49231ea56fe1d0e2e"
format_stamp: "Formatted at 2026-09-28 08:38:34 on dist-test-slave-w6c0"
server_key: "c49fd610f6e1af0afbcac3317aa57ec6"
server_key_iv: "8287ea0573ddaaf3584e5f74a0733da4"
server_key_version: "encryptionkey@0"
I20260928 08:38:34.697728 10702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260928 08:38:34.701128 10717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:34.702330 10702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:38:34.702580 10702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "b7be6582fb6648e49231ea56fe1d0e2e"
format_stamp: "Formatted at 2026-09-28 08:38:34 on dist-test-slave-w6c0"
server_key: "c49fd610f6e1af0afbcac3317aa57ec6"
server_key_iv: "8287ea0573ddaaf3584e5f74a0733da4"
server_key_version: "encryptionkey@0"
I20260928 08:38:34.702773 10702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:34.718374 10702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:34.722353 10702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:34.722679 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:34.740870 10702 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35051
I20260928 08:38:34.740890 10768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35051 every 8 connection(s)
I20260928 08:38:34.742313 10702 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:38:34.746022 10769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:34.752211 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 10702
I20260928 08:38:34.752449 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:38:34.753278 388 external_mini_cluster.cc:1428] Setting key eeb5fc3adccb8520d1e0e91b508f54ec
I20260928 08:38:34.757428 10769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e: Bootstrap starting.
I20260928 08:38:34.761363 10769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:34.762530 10769 log.cc:824] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:38:34.765604 10769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e: No bootstrap required, opened a new log
Sep 28 08:38:34 dist-test-slave-w6c0 krb5kdc[10689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:38:34.774188 10769 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7be6582fb6648e49231ea56fe1d0e2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35051 } }
I20260928 08:38:34.774600 10769 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:34.774737 10769 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7be6582fb6648e49231ea56fe1d0e2e, State: Initialized, Role: FOLLOWER
I20260928 08:38:34.775435 10769 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7be6582fb6648e49231ea56fe1d0e2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35051 } }
I20260928 08:38:34.775686 10769 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:38:34.775822 10769 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:38:34.776072 10769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:34.776947 10772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:34.755502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43938 (local address 127.0.97.62:35051)
0928 08:38:34.756281 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:34.756292 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:34.756333 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:38:34.758111 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:34.758134 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:34.758584 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:34.758921 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:34.760836 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:34.762275 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:34.763903 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:34.764547 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:34.764672 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:34.769205 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:34.769213 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:34.769228 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:34.769268 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:34.772517 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:34.773249 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:34.773255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:34.773261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:34.773434 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:34.773803 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:34.773812 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:34.773815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:34.774325 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:34.774646 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:34.775106 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 08:38:34.775534 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":373,"threads_started":1}
I20260928 08:38:34.779120 10769 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7be6582fb6648e49231ea56fe1d0e2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35051 } }
I20260928 08:38:34.779668 10769 leader_election.cc:302] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7be6582fb6648e49231ea56fe1d0e2e; no voters:
I20260928 08:38:34.780267 10769 leader_election.cc:288] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:38:34.780418 10774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:38:34.781092 10774 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e [term 1 LEADER]: Becoming Leader. State: Replica: b7be6582fb6648e49231ea56fe1d0e2e, State: Running, Role: LEADER
I20260928 08:38:34.781594 10774 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7be6582fb6648e49231ea56fe1d0e2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35051 } }
I20260928 08:38:34.782195 10769 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:38:34.784307 10776 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7be6582fb6648e49231ea56fe1d0e2e. Latest consensus state: current_term: 1 leader_uuid: "b7be6582fb6648e49231ea56fe1d0e2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7be6582fb6648e49231ea56fe1d0e2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35051 } } }
I20260928 08:38:34.784644 10776 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e [sys.catalog]: This master's current role is: LEADER
I20260928 08:38:34.785432 10779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:38:34.786489 10775 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7be6582fb6648e49231ea56fe1d0e2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7be6582fb6648e49231ea56fe1d0e2e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35051 } } }
I20260928 08:38:34.786854 10775 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e [sys.catalog]: This master's current role is: LEADER
I20260928 08:38:34.790813 10779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:38:34.800064 10779 catalog_manager.cc:1347] Generated new cluster ID: 448224fec15a4114bf2ecc7ff4c5c2e6
I20260928 08:38:34.800177 10779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:38:34.815688 10779 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:38:34.816994 10779 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:38:34.826428 10779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e: Generated new TSK 0
I20260928 08:38:34.827390 10779 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:38:34.888329 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35051
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:38:35.218987 10795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:35.219256 10795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:35.219350 10795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:35.226732 10795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:35.226835 10795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:35.226953 10795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:38:35.236585 10795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:35.237780 10795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:35.239037 10795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:35.251379 10801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:35.251513 10800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:35.251479 10803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:35.252579 10795 server_base.cc:1034] running on GCE node
I20260928 08:38:35.253173 10795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:35.254060 10795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:35.255275 10795 hybrid_clock.cc:648] HybridClock initialized: now 1790584715255225 us; error 64 us; skew 500 ppm
I20260928 08:38:35.255568 10795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:38:35 dist-test-slave-w6c0 krb5kdc[10689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:38:35.258580 10795 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:38:35.259963 10795 webserver.cc:460] Webserver started at http://127.0.97.1:36489/ using document root <none> and password file <none>
I20260928 08:38:35.260586 10795 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:35.260677 10795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:35.260916 10795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:35.263742 10795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "996b038dd90e4e63956a5e99973cb44d"
format_stamp: "Formatted at 2026-09-28 08:38:35 on dist-test-slave-w6c0"
server_key: "79c52c4c4826e83041556c3c377b4b45"
server_key_iv: "1bbadf45fc9e33d3948163d89dd5cc22"
server_key_version: "encryptionkey@0"
I20260928 08:38:35.264430 10795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "996b038dd90e4e63956a5e99973cb44d"
format_stamp: "Formatted at 2026-09-28 08:38:35 on dist-test-slave-w6c0"
server_key: "79c52c4c4826e83041556c3c377b4b45"
server_key_iv: "1bbadf45fc9e33d3948163d89dd5cc22"
server_key_version: "encryptionkey@0"
I20260928 08:38:35.269424 10795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260928 08:38:35.272712 10810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:35.273957 10795 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 08:38:35.274121 10795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "996b038dd90e4e63956a5e99973cb44d"
format_stamp: "Formatted at 2026-09-28 08:38:35 on dist-test-slave-w6c0"
server_key: "79c52c4c4826e83041556c3c377b4b45"
server_key_iv: "1bbadf45fc9e33d3948163d89dd5cc22"
server_key_version: "encryptionkey@0"
I20260928 08:38:35.274274 10795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:35.296882 10795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:35.300961 10795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:35.301239 10795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:35.302192 10795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:38:35.303646 10795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:38:35.303720 10795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:35.303839 10795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:38:35.303902 10795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:35.337247 10795 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45961
I20260928 08:38:35.337324 10922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45961 every 8 connection(s)
I20260928 08:38:35.338578 10795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:38:35.343923 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 10795
I20260928 08:38:35.344143 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:38:35.344918 388 external_mini_cluster.cc:1428] Setting key 53ef0666620cc21a6b7f46161d51616f
Sep 28 08:38:35 dist-test-slave-w6c0 krb5kdc[10689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:38:35.359366 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:35.341353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34129 (local address 127.0.97.62:35051)
0928 08:38:35.341795 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:35.341799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:35.342573 (+ 774us) server_negotiation.cc:407] Connection header received
0928 08:38:35.344831 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:35.344835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:35.344915 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:35.345045 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:35.346813 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:35.347873 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:35.349381 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:35.350061 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:35.350200 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:35.353443 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:35.353452 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:35.353460 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:35.353503 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:35.356252 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:35.357183 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:35.357186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:35.357189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:35.357296 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:35.357741 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:35.357744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:35.357746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:35.357980 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:35.358081 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:35.358960 (+ 879us) server_negotiation.cc:299] Negotiation successful
0928 08:38:35.359114 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":244,"threads_started":1}
I20260928 08:38:35.360857 10926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:35.341717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35051 (local address 127.0.97.1:34129)
0928 08:38:35.342397 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:35.342437 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:35.344539 (+ 2102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:35.345295 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:35.345307 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:35.345869 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:35.346651 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:35.346676 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:35.348032 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:35.348036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:35.349231 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:35.349242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:35.350255 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:35.350260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:35.350442 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:35.351154 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:35.351175 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:35.353238 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:38:35.356399 (+ 3161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:35.356411 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:35.356429 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:35.357066 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:35.357398 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:35.357405 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:35.357412 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:35.357635 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:35.358104 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:38:35.358116 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:38:35.358578 (+ 462us) client_negotiation.cc:769] Sending connection context
0928 08:38:35.358866 (+ 288us) client_negotiation.cc:240] Negotiation successful
0928 08:38:35.359420 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":271,"threads_started":1}
I20260928 08:38:35.362358 10923 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35051
I20260928 08:38:35.362671 10923 heartbeater.cc:464] Registering TS with master...
I20260928 08:38:35.363324 10923 heartbeater.cc:511] Master 127.0.97.62:35051 requested a full tablet report, sending...
I20260928 08:38:35.365298 10734 ts_manager.cc:194] Registered new tserver with Master: 996b038dd90e4e63956a5e99973cb44d (127.0.97.1:45961)
I20260928 08:38:35.366803 10734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34129
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:38:35.397764 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35051
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:38:35.714388 10929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:35.714782 10929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:35.714895 10929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:35.722198 10929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:35.722311 10929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:35.722429 10929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:38:35.732381 10929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:35.733626 10929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:35.734956 10929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:35.748692 10934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:35.748858 10935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:35.749073 10937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:35.749826 10929 server_base.cc:1034] running on GCE node
I20260928 08:38:35.750428 10929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:35.751394 10929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:35.752635 10929 hybrid_clock.cc:648] HybridClock initialized: now 1790584715752576 us; error 48 us; skew 500 ppm
I20260928 08:38:35.752969 10929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:38:35 dist-test-slave-w6c0 krb5kdc[10689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:38:35.756271 10929 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:38:35.757898 10929 webserver.cc:460] Webserver started at http://127.0.97.2:35315/ using document root <none> and password file <none>
I20260928 08:38:35.758713 10929 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:35.758857 10929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:35.759152 10929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:35.762084 10929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "ea115261f20f4d90ac93de5f994e9122"
format_stamp: "Formatted at 2026-09-28 08:38:35 on dist-test-slave-w6c0"
server_key: "02f8c6ad676cee4d8b3b7057fd4133a8"
server_key_iv: "04e828391c0b4dc5d19fc1989974608e"
server_key_version: "encryptionkey@0"
I20260928 08:38:35.762924 10929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ea115261f20f4d90ac93de5f994e9122"
format_stamp: "Formatted at 2026-09-28 08:38:35 on dist-test-slave-w6c0"
server_key: "02f8c6ad676cee4d8b3b7057fd4133a8"
server_key_iv: "04e828391c0b4dc5d19fc1989974608e"
server_key_version: "encryptionkey@0"
I20260928 08:38:35.768213 10929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:38:35.771694 10944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:35.773322 10929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 08:38:35.773500 10929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "ea115261f20f4d90ac93de5f994e9122"
format_stamp: "Formatted at 2026-09-28 08:38:35 on dist-test-slave-w6c0"
server_key: "02f8c6ad676cee4d8b3b7057fd4133a8"
server_key_iv: "04e828391c0b4dc5d19fc1989974608e"
server_key_version: "encryptionkey@0"
I20260928 08:38:35.773700 10929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:35.808632 10929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:35.812518 10929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:35.812804 10929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:35.813751 10929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:38:35.815100 10929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:38:35.815191 10929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:35.815361 10929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:38:35.815411 10929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:35.847708 10929 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39197
I20260928 08:38:35.847817 11056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39197 every 8 connection(s)
I20260928 08:38:35.848985 10929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:38:35.857998 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 10929
I20260928 08:38:35.858196 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:38:35.858687 388 external_mini_cluster.cc:1428] Setting key 28d2ec874d46c467a1115a7dd76b1982
Sep 28 08:38:35 dist-test-slave-w6c0 krb5kdc[10689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:38:35.868949 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:35.851455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48943 (local address 127.0.97.62:35051)
0928 08:38:35.851619 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:35.851623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:35.852666 (+ 1043us) server_negotiation.cc:407] Connection header received
0928 08:38:35.854307 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:35.854312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:35.854379 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:35.854467 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:35.855884 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:35.856562 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:35.858063 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:35.858621 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:35.858756 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:35.863101 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:35.863106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:35.863109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:35.863139 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:35.865578 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:35.866543 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:35.866550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:35.866560 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:35.866678 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:35.867162 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:35.867166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:35.867168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:35.867399 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:35.867503 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:35.868471 (+ 968us) server_negotiation.cc:299] Negotiation successful
0928 08:38:35.868707 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 08:38:35.870203 11059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:35.851847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35051 (local address 127.0.97.2:48943)
0928 08:38:35.852494 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:35.852535 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:35.854069 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:35.854713 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:35.854720 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:35.855200 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:35.855725 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:35.855745 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:35.856695 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:35.856698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:35.857913 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:35.857924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:35.858839 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:35.858846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:35.859067 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:35.859876 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:35.859903 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:35.862894 (+ 2991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:38:35.865727 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:35.865738 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:35.865757 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:35.866387 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:35.866786 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:35.866793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:35.866800 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:35.867058 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:35.867520 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:38:35.867530 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:38:35.867977 (+ 447us) client_negotiation.cc:769] Sending connection context
0928 08:38:35.868285 (+ 308us) client_negotiation.cc:240] Negotiation successful
0928 08:38:35.868864 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":254,"threads_started":1}
I20260928 08:38:35.871649 11057 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35051
I20260928 08:38:35.871989 11057 heartbeater.cc:464] Registering TS with master...
I20260928 08:38:35.872817 11057 heartbeater.cc:511] Master 127.0.97.62:35051 requested a full tablet report, sending...
I20260928 08:38:35.874321 10734 ts_manager.cc:194] Registered new tserver with Master: ea115261f20f4d90ac93de5f994e9122 (127.0.97.2:39197)
I20260928 08:38:35.875172 10734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48943
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:38:35.909437 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35051
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:38:36.231698 11062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:38:36.232023 11062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:38:36.232187 11062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:38:36.239391 11062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:38:36.239550 11062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:38:36.239733 11062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:38:36.249266 11062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:38:36.250577 11062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:38:36.251864 11062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:38:36.264940 11070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:36.265060 11067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:38:36.265040 11068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:38:36.265844 11062 server_base.cc:1034] running on GCE node
I20260928 08:38:36.266426 11062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:38:36.267334 11062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:38:36.268580 11062 hybrid_clock.cc:648] HybridClock initialized: now 1790584716268538 us; error 56 us; skew 500 ppm
I20260928 08:38:36.268904 11062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:38:36 dist-test-slave-w6c0 krb5kdc[10689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:38:36.271953 11062 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:38:36.273483 11062 webserver.cc:460] Webserver started at http://127.0.97.3:44473/ using document root <none> and password file <none>
I20260928 08:38:36.274199 11062 fs_manager.cc:359] Metadata directory not provided
I20260928 08:38:36.274345 11062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:38:36.274732 11062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:38:36.277822 11062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "ff9915f64d4748a5b5e2c29354bd8ef6"
format_stamp: "Formatted at 2026-09-28 08:38:36 on dist-test-slave-w6c0"
server_key: "9d51b030620942b040dad2d080767422"
server_key_iv: "3f3a2154b4add003cbf6f1c88eeaa604"
server_key_version: "encryptionkey@0"
I20260928 08:38:36.278725 11062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ff9915f64d4748a5b5e2c29354bd8ef6"
format_stamp: "Formatted at 2026-09-28 08:38:36 on dist-test-slave-w6c0"
server_key: "9d51b030620942b040dad2d080767422"
server_key_iv: "3f3a2154b4add003cbf6f1c88eeaa604"
server_key_version: "encryptionkey@0"
I20260928 08:38:36.284160 11062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:38:36.287654 11077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:36.288960 11062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:38:36.289129 11062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "ff9915f64d4748a5b5e2c29354bd8ef6"
format_stamp: "Formatted at 2026-09-28 08:38:36 on dist-test-slave-w6c0"
server_key: "9d51b030620942b040dad2d080767422"
server_key_iv: "3f3a2154b4add003cbf6f1c88eeaa604"
server_key_version: "encryptionkey@0"
I20260928 08:38:36.289335 11062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:38:36.304111 11062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:38:36.308013 11062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:38:36.308323 11062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:38:36.309276 11062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:38:36.310647 11062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:38:36.310719 11062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:36.310801 11062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:38:36.310848 11062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:38:36.344238 11062 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40447
I20260928 08:38:36.344255 11189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40447 every 8 connection(s)
I20260928 08:38:36.345659 11062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:38:36.349869 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 11062
I20260928 08:38:36.350091 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:38:36.350564 388 external_mini_cluster.cc:1428] Setting key b77b9a1a4823689a6af0f8faaa5c5e08
Sep 28 08:38:36 dist-test-slave-w6c0 krb5kdc[10689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:38:36.364773 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:36.348252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40685 (local address 127.0.97.62:35051)
0928 08:38:36.348464 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:36.348468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:36.349680 (+ 1212us) server_negotiation.cc:407] Connection header received
0928 08:38:36.351728 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:36.351733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:36.351819 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:36.351939 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:36.353331 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.354110 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.355646 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.356045 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.356178 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:36.359501 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:36.359506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:36.359509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:36.359550 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:36.361848 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:36.362500 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:36.362554 (+ 54us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:36.362558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:36.362686 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:36.363047 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:36.363050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:36.363052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:36.363280 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:36.363386 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:36.364223 (+ 837us) server_negotiation.cc:299] Negotiation successful
0928 08:38:36.364434 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 08:38:36.365701 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:36.348719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35051 (local address 127.0.97.3:40685)
0928 08:38:36.349512 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:36.349556 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:36.351480 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:36.352159 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:36.352167 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:36.352647 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:38:36.353174 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:36.353189 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.354263 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.354267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:36.355507 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:36.355518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.356261 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.356270 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:36.356494 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:36.357105 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:38:36.357126 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:38:36.359313 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:38:36.361981 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:36.361987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:36.361998 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:36.362383 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:36.362779 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:38:36.362781 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:38:36.362784 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:38:36.362950 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:38:36.363383 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:38:36.363428 (+ 45us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:38:36.363817 (+ 389us) client_negotiation.cc:769] Sending connection context
0928 08:38:36.364069 (+ 252us) client_negotiation.cc:240] Negotiation successful
0928 08:38:36.364566 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":366,"threads_started":1}
I20260928 08:38:36.367332 11190 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35051
I20260928 08:38:36.367637 11190 heartbeater.cc:464] Registering TS with master...
I20260928 08:38:36.368345 11190 heartbeater.cc:511] Master 127.0.97.62:35051 requested a full tablet report, sending...
I20260928 08:38:36.369674 10923 heartbeater.cc:503] Master 127.0.97.62:35051 was elected leader, sending a full tablet report...
I20260928 08:38:36.369839 10734 ts_manager.cc:194] Registered new tserver with Master: ff9915f64d4748a5b5e2c29354bd8ef6 (127.0.97.3:40447)
I20260928 08:38:36.370860 10734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40685
I20260928 08:38:36.377533 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:38:36.393687 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:36.381888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.0.97.62:35051)
0928 08:38:36.382055 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:36.382059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:36.382611 (+ 552us) server_negotiation.cc:407] Connection header received
0928 08:38:36.382752 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:36.382756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:36.382815 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:36.382896 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:36.384179 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.384873 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.386026 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.386384 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.386469 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:36.388596 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:36.388601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:36.388604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:36.388633 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:36.391275 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:36.391863 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:36.391867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:36.391870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:36.391987 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:36.392404 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:36.392407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:36.392409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:36.392699 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:36.393029 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:36.393250 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 08:38:36.393410 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 08:38:36.398444 10734 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:38:36.405130 10734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:38:36.441103 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:36.417992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45961 (local address 127.0.0.1:49880)
0928 08:38:36.432437 (+ 14445us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:36.432453 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:36.432585 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:36.433274 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:36.433282 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:36.433313 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:38:36.433768 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:36.433782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.435792 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.435798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:36.437940 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:36.437951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.440005 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.440012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:36.440223 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:36.440272 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 08:38:36.440409 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 08:38:36.440609 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14338,"mutex_wait_us":104,"thread_start_us":14357,"threads_started":1}
I20260928 08:38:36.441350 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:36.417402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54236 (local address 127.0.97.2:39197)
0928 08:38:36.432605 (+ 15203us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:36.432616 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:36.432643 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:38:36.432739 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:36.432748 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:36.433028 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:36.434085 (+ 1057us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:38:36.434769 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.435792 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.438594 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.440035 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.440158 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:36.440335 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:36.440856 (+ 521us) server_negotiation.cc:299] Negotiation successful
0928 08:38:36.441037 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15045,"thread_start_us":16291,"threads_started":1}
I20260928 08:38:36.441732 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:36.416881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34486 (local address 127.0.97.3:40447)
0928 08:38:36.432969 (+ 16088us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:36.432977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:36.433007 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:38:36.433119 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:36.433124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:36.433353 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:36.433549 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:38:36.434504 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.436278 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.438868 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.440162 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.440293 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:36.440516 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:36.441211 (+ 695us) server_negotiation.cc:299] Negotiation successful
0928 08:38:36.441367 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15917,"thread_start_us":16541,"threads_started":1}
I20260928 08:38:36.441864 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:36.418391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.0.97.1:45961)
0928 08:38:36.426798 (+ 8407us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:36.426812 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:36.432551 (+ 5739us) server_negotiation.cc:407] Connection header received
0928 08:38:36.432739 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:36.432748 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:36.433016 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:36.433257 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:38:36.433969 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.435642 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.438108 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.439478 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.439606 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:36.439834 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:36.441369 (+ 1535us) server_negotiation.cc:299] Negotiation successful
0928 08:38:36.441545 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8182,"thread_start_us":7951,"threads_started":1}
I20260928 08:38:36.441593 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:36.417121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39197 (local address 127.0.0.1:54236)
0928 08:38:36.432010 (+ 14889us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:36.432048 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:36.432225 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:36.434084 (+ 1859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:36.434089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:36.434116 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:38:36.434601 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:36.434610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.435947 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.435952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:36.438423 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:36.438437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.440482 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.440488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:36.440655 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:36.440682 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 08:38:36.440771 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 08:38:36.440888 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14691,"mutex_wait_us":60,"thread_start_us":15546,"threads_started":1}
I20260928 08:38:36.441843 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:36.416697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40447 (local address 127.0.0.1:34486)
0928 08:38:36.417569 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:36.417611 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:36.417766 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:36.433849 (+ 16083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:36.433856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:36.433883 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:38:36.434312 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:36.434323 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.436439 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.436444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:36.438726 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:36.438736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.440596 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.440606 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:36.440718 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:36.440766 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 08:38:36.440853 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 08:38:36.440989 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":566,"threads_started":1}
I20260928 08:38:36.445647 10992 tablet_service.cc:1467] Processing CreateTablet for tablet 20c509ebff3a4c1a97c608a061cad668 (DEFAULT_TABLE table=test-table [id=1a055824048642369bddd9c405d00163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:38:36.445811 11125 tablet_service.cc:1467] Processing CreateTablet for tablet 20c509ebff3a4c1a97c608a061cad668 (DEFAULT_TABLE table=test-table [id=1a055824048642369bddd9c405d00163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:38:36.445839 10858 tablet_service.cc:1467] Processing CreateTablet for tablet 20c509ebff3a4c1a97c608a061cad668 (DEFAULT_TABLE table=test-table [id=1a055824048642369bddd9c405d00163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:38:36.447084 10992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c509ebff3a4c1a97c608a061cad668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:36.447137 10858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c509ebff3a4c1a97c608a061cad668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:36.447134 11125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c509ebff3a4c1a97c608a061cad668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:38:36.458168 11208 tablet_bootstrap.cc:492] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122: Bootstrap starting.
I20260928 08:38:36.460999 11209 tablet_bootstrap.cc:492] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d: Bootstrap starting.
I20260928 08:38:36.461755 11210 tablet_bootstrap.cc:492] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6: Bootstrap starting.
I20260928 08:38:36.462551 11208 tablet_bootstrap.cc:654] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:36.464246 11208 log.cc:824] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122: Log is configured to *not* fsync() on all Append() calls
I20260928 08:38:36.465003 11209 tablet_bootstrap.cc:654] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:36.465126 11210 tablet_bootstrap.cc:654] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6: Neither blocks nor log segments found. Creating new log.
I20260928 08:38:36.466051 11209 log.cc:824] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:38:36.466194 11210 log.cc:824] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6: Log is configured to *not* fsync() on all Append() calls
I20260928 08:38:36.467386 11208 tablet_bootstrap.cc:492] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122: No bootstrap required, opened a new log
I20260928 08:38:36.467648 11208 ts_tablet_manager.cc:1397] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 08:38:36.468575 11209 tablet_bootstrap.cc:492] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d: No bootstrap required, opened a new log
I20260928 08:38:36.468809 11209 ts_tablet_manager.cc:1397] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260928 08:38:36.472478 11210 tablet_bootstrap.cc:492] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6: No bootstrap required, opened a new log
I20260928 08:38:36.472776 11210 ts_tablet_manager.cc:1397] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260928 08:38:36.475474 11208 raft_consensus.cc:348] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9915f64d4748a5b5e2c29354bd8ef6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40447 } } peers { permanent_uuid: "ea115261f20f4d90ac93de5f994e9122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39197 } } peers { permanent_uuid: "996b038dd90e4e63956a5e99973cb44d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } }
I20260928 08:38:36.475879 11208 raft_consensus.cc:374] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:36.475996 11208 raft_consensus.cc:729] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea115261f20f4d90ac93de5f994e9122, State: Initialized, Role: FOLLOWER
I20260928 08:38:36.476620 11208 consensus_queue.cc:260] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9915f64d4748a5b5e2c29354bd8ef6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40447 } } peers { permanent_uuid: "ea115261f20f4d90ac93de5f994e9122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39197 } } peers { permanent_uuid: "996b038dd90e4e63956a5e99973cb44d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } }
I20260928 08:38:36.477854 11057 heartbeater.cc:503] Master 127.0.97.62:35051 was elected leader, sending a full tablet report...
I20260928 08:38:36.478754 11208 ts_tablet_manager.cc:1428] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122: Time spent starting tablet: real 0.011s user 0.006s sys 0.006s
I20260928 08:38:36.479640 11209 raft_consensus.cc:348] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9915f64d4748a5b5e2c29354bd8ef6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40447 } } peers { permanent_uuid: "ea115261f20f4d90ac93de5f994e9122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39197 } } peers { permanent_uuid: "996b038dd90e4e63956a5e99973cb44d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } }
I20260928 08:38:36.480011 11209 raft_consensus.cc:374] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:36.480123 11209 raft_consensus.cc:729] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b038dd90e4e63956a5e99973cb44d, State: Initialized, Role: FOLLOWER
I20260928 08:38:36.480679 11209 consensus_queue.cc:260] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9915f64d4748a5b5e2c29354bd8ef6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40447 } } peers { permanent_uuid: "ea115261f20f4d90ac93de5f994e9122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39197 } } peers { permanent_uuid: "996b038dd90e4e63956a5e99973cb44d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } }
I20260928 08:38:36.482491 11209 ts_tablet_manager.cc:1428] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260928 08:38:36.482913 11210 raft_consensus.cc:348] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9915f64d4748a5b5e2c29354bd8ef6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40447 } } peers { permanent_uuid: "ea115261f20f4d90ac93de5f994e9122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39197 } } peers { permanent_uuid: "996b038dd90e4e63956a5e99973cb44d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } }
I20260928 08:38:36.483342 11210 raft_consensus.cc:374] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:38:36.483438 11210 raft_consensus.cc:729] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9915f64d4748a5b5e2c29354bd8ef6, State: Initialized, Role: FOLLOWER
I20260928 08:38:36.483995 11210 consensus_queue.cc:260] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9915f64d4748a5b5e2c29354bd8ef6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40447 } } peers { permanent_uuid: "ea115261f20f4d90ac93de5f994e9122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39197 } } peers { permanent_uuid: "996b038dd90e4e63956a5e99973cb44d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } }
I20260928 08:38:36.486987 11190 heartbeater.cc:503] Master 127.0.97.62:35051 was elected leader, sending a full tablet report...
I20260928 08:38:36.487480 11210 ts_tablet_manager.cc:1428] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
W20260928 08:38:36.591090 10924 tablet.cc:2369] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:38:36.597340 11191 tablet.cc:2369] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:38:36.600906 11058 tablet.cc:2369] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:38:36.628885 11216 raft_consensus.cc:482] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:38:36.629194 11216 raft_consensus.cc:504] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9915f64d4748a5b5e2c29354bd8ef6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40447 } } peers { permanent_uuid: "ea115261f20f4d90ac93de5f994e9122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39197 } } peers { permanent_uuid: "996b038dd90e4e63956a5e99973cb44d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } }
I20260928 08:38:36.630882 11216 leader_election.cc:288] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea115261f20f4d90ac93de5f994e9122 (127.0.97.2:39197), 996b038dd90e4e63956a5e99973cb44d (127.0.97.1:45961)
I20260928 08:38:36.637358 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:36.631542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51619 (local address 127.0.97.1:45961)
0928 08:38:36.631814 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:36.631818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:36.631833 (+ 15us) server_negotiation.cc:407] Connection header received
0928 08:38:36.631889 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:36.631893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:36.631947 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:36.632060 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:38:36.632508 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.633521 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.635382 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.636491 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.636590 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:36.636665 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:36.637060 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 08:38:36.637164 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260928 08:38:36.637400 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:36.631212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45961 (local address 127.0.97.3:51619)
0928 08:38:36.631500 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:36.631523 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:36.631660 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:36.632069 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:36.632074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:36.632100 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:38:36.632346 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:36.632358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.633663 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.633669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:36.635226 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:36.635237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.636649 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.636655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:36.636848 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:36.636884 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 08:38:36.636986 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 08:38:36.637121 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260928 08:38:36.638470 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:36.631408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54255 (local address 127.0.97.2:39197)
0928 08:38:36.631615 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:36.631621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:36.632273 (+ 652us) server_negotiation.cc:407] Connection header received
0928 08:38:36.632406 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:36.632413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:36.632500 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:36.632566 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:38:36.633236 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.634234 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.636241 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.637367 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.637507 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:36.637614 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:36.638091 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 08:38:36.638220 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:38:36.638536 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:36.631290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39197 (local address 127.0.97.3:54255)
0928 08:38:36.632158 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0928 08:38:36.632173 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:38:36.632279 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:38:36.632647 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:38:36.632654 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:38:36.632680 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:38:36.633006 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:36.633017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.634387 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.634390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:36.636075 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:38:36.636090 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.637741 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.637748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:38:36.637918 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:36.637949 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 08:38:36.638066 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 08:38:36.638195 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":531,"threads_started":1}
I20260928 08:38:36.638335 10878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c509ebff3a4c1a97c608a061cad668" candidate_uuid: "ff9915f64d4748a5b5e2c29354bd8ef6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b038dd90e4e63956a5e99973cb44d" is_pre_election: true
I20260928 08:38:36.638947 10878 raft_consensus.cc:2454] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff9915f64d4748a5b5e2c29354bd8ef6 in term 0.
I20260928 08:38:36.639605 11080 leader_election.cc:302] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996b038dd90e4e63956a5e99973cb44d, ff9915f64d4748a5b5e2c29354bd8ef6; no voters:
I20260928 08:38:36.639529 11012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c509ebff3a4c1a97c608a061cad668" candidate_uuid: "ff9915f64d4748a5b5e2c29354bd8ef6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea115261f20f4d90ac93de5f994e9122" is_pre_election: true
I20260928 08:38:36.639993 11216 raft_consensus.cc:2781] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:38:36.640002 11012 raft_consensus.cc:2454] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff9915f64d4748a5b5e2c29354bd8ef6 in term 0.
I20260928 08:38:36.640117 11216 raft_consensus.cc:482] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:38:36.640247 11216 raft_consensus.cc:3037] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:36.642439 11216 raft_consensus.cc:504] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9915f64d4748a5b5e2c29354bd8ef6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40447 } } peers { permanent_uuid: "ea115261f20f4d90ac93de5f994e9122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39197 } } peers { permanent_uuid: "996b038dd90e4e63956a5e99973cb44d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } }
I20260928 08:38:36.643118 11216 leader_election.cc:288] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [CANDIDATE]: Term 1 election: Requested vote from peers ea115261f20f4d90ac93de5f994e9122 (127.0.97.2:39197), 996b038dd90e4e63956a5e99973cb44d (127.0.97.1:45961)
I20260928 08:38:36.643540 11012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c509ebff3a4c1a97c608a061cad668" candidate_uuid: "ff9915f64d4748a5b5e2c29354bd8ef6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea115261f20f4d90ac93de5f994e9122"
I20260928 08:38:36.643594 10878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c509ebff3a4c1a97c608a061cad668" candidate_uuid: "ff9915f64d4748a5b5e2c29354bd8ef6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b038dd90e4e63956a5e99973cb44d"
I20260928 08:38:36.643746 11012 raft_consensus.cc:3037] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:36.643782 10878 raft_consensus.cc:3037] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:38:36.645859 11012 raft_consensus.cc:2454] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9915f64d4748a5b5e2c29354bd8ef6 in term 1.
I20260928 08:38:36.645861 10878 raft_consensus.cc:2454] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9915f64d4748a5b5e2c29354bd8ef6 in term 1.
I20260928 08:38:36.646279 11080 leader_election.cc:302] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996b038dd90e4e63956a5e99973cb44d, ff9915f64d4748a5b5e2c29354bd8ef6; no voters:
I20260928 08:38:36.646631 11216 raft_consensus.cc:2781] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:38:36.647051 11216 raft_consensus.cc:686] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [term 1 LEADER]: Becoming Leader. State: Replica: ff9915f64d4748a5b5e2c29354bd8ef6, State: Running, Role: LEADER
I20260928 08:38:36.647507 11216 consensus_queue.cc:237] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9915f64d4748a5b5e2c29354bd8ef6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40447 } } peers { permanent_uuid: "ea115261f20f4d90ac93de5f994e9122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39197 } } peers { permanent_uuid: "996b038dd90e4e63956a5e99973cb44d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } }
I20260928 08:38:36.652258 10733 catalog_manager.cc:5495] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff9915f64d4748a5b5e2c29354bd8ef6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ff9915f64d4748a5b5e2c29354bd8ef6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9915f64d4748a5b5e2c29354bd8ef6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea115261f20f4d90ac93de5f994e9122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "996b038dd90e4e63956a5e99973cb44d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } health_report { overall_health: UNKNOWN } } }
I20260928 08:38:36.681509 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:36.675689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34498 (local address 127.0.97.3:40447)
0928 08:38:36.675915 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:36.675923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:36.675945 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:38:36.676193 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:36.676197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:36.676276 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:36.676434 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:38:36.676836 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.677651 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.679323 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.679753 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.679889 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:36.680441 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:38:36.680947 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:38:36.681116 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:36.681166 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 08:38:36.681244 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 08:38:36.689774 10878 raft_consensus.cc:1261] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d [term 1 FOLLOWER]: Refusing update from remote peer ff9915f64d4748a5b5e2c29354bd8ef6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:38:36.689759 11012 raft_consensus.cc:1261] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122 [term 1 FOLLOWER]: Refusing update from remote peer ff9915f64d4748a5b5e2c29354bd8ef6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:38:36.690896 11218 consensus_queue.cc:1035] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "996b038dd90e4e63956a5e99973cb44d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:38:36.691287 11216 consensus_queue.cc:1035] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea115261f20f4d90ac93de5f994e9122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:38:36.702955 11221 mvcc.cc:205] Tried to move back new op lower bound from 7334234999550345216 to 7334234999391035392. Current Snapshot: MvccSnapshot[applied={T|T < 7334234999550345216}]
I20260928 08:38:36.717285 10733 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:38:36.717600 10733 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:38:36.721392 10733 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7be6582fb6648e49231ea56fe1d0e2e: Sending DeleteTablet for 3 replicas of tablet 20c509ebff3a4c1a97c608a061cad668
I20260928 08:38:36.722599 10992 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c509ebff3a4c1a97c608a061cad668 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:38:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54236
I20260928 08:38:36.722643 11125 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c509ebff3a4c1a97c608a061cad668 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:38:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34486
I20260928 08:38:36.722793 10858 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c509ebff3a4c1a97c608a061cad668 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:38:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49880
I20260928 08:38:36.724941 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 10795
I20260928 08:38:36.738163 11231 tablet_replica.cc:330] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6: stopping tablet replica
I20260928 08:38:36.738890 11231 raft_consensus.cc:2229] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:38:36.739789 11231 raft_consensus.cc:2258] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:38:36.756348 11232 tablet_replica.cc:330] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d: stopping tablet replica
I20260928 08:38:36.756912 11232 raft_consensus.cc:2229] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:38:36.757537 11232 raft_consensus.cc:2258] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:38:36.758365 11230 tablet_replica.cc:330] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122: stopping tablet replica
I20260928 08:38:36.759975 11231 ts_tablet_manager.cc:1905] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:38:36.760556 11230 raft_consensus.cc:2229] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:38:36.761080 11230 raft_consensus.cc:2258] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:38:36.764246 11230 ts_tablet_manager.cc:1905] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 08:38:36 dist-test-slave-w6c0 krb5kdc[10689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:38:36.767979 11231 ts_tablet_manager.cc:1918] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:38:36.768292 11231 log.cc:1196] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/20c509ebff3a4c1a97c608a061cad668
I20260928 08:38:36.768958 11231 ts_tablet_manager.cc:1939] T 20c509ebff3a4c1a97c608a061cad668 P ff9915f64d4748a5b5e2c29354bd8ef6: Deleting consensus metadata
I20260928 08:38:36.770941 10720 catalog_manager.cc:4841] TS ff9915f64d4748a5b5e2c29354bd8ef6 (127.0.97.3:40447): tablet 20c509ebff3a4c1a97c608a061cad668 (table test-table [id=1a055824048642369bddd9c405d00163]) successfully deleted
I20260928 08:38:36.771019 11232 ts_tablet_manager.cc:1905] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:38:36.772106 11230 ts_tablet_manager.cc:1918] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:38:36.772286 11230 log.cc:1196] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/20c509ebff3a4c1a97c608a061cad668
I20260928 08:38:36.772818 11230 ts_tablet_manager.cc:1939] T 20c509ebff3a4c1a97c608a061cad668 P ea115261f20f4d90ac93de5f994e9122: Deleting consensus metadata
I20260928 08:38:36.774618 10719 catalog_manager.cc:4841] TS ea115261f20f4d90ac93de5f994e9122 (127.0.97.2:39197): tablet 20c509ebff3a4c1a97c608a061cad668 (table test-table [id=1a055824048642369bddd9c405d00163]) successfully deleted
I20260928 08:38:36.776158 11232 ts_tablet_manager.cc:1918] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:38:36.776274 11232 log.cc:1196] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584618551208-388-0/minicluster-data/ts-0/wal/wals/20c509ebff3a4c1a97c608a061cad668
I20260928 08:38:36.776680 11232 ts_tablet_manager.cc:1939] T 20c509ebff3a4c1a97c608a061cad668 P 996b038dd90e4e63956a5e99973cb44d: Deleting consensus metadata
I20260928 08:38:36.778209 10720 catalog_manager.cc:4841] TS 996b038dd90e4e63956a5e99973cb44d (127.0.97.1:45961): tablet 20c509ebff3a4c1a97c608a061cad668 (table test-table [id=1a055824048642369bddd9c405d00163]) successfully deleted
I20260928 08:38:36.781826 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:36.727898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.0.97.1:45961)
0928 08:38:36.728123 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:36.728132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:36.750856 (+ 22724us) server_negotiation.cc:407] Connection header received
0928 08:38:36.751107 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:36.751113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:36.751196 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:36.751270 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:36.752940 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.754114 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.758588 (+ 4474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.759203 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.759304 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:36.765633 (+ 6329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:36.765652 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:36.765686 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:36.765739 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:36.769702 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:36.779453 (+ 9751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:36.779463 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:36.779467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:36.779648 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:36.780087 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:36.780092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:36.780095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:36.780609 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:36.780904 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:36.781214 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 08:38:36.781452 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:38:36.782745 10918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:36.872856 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 10795
I20260928 08:38:36.892047 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 10929
Sep 28 08:38:36 dist-test-slave-w6c0 krb5kdc[10689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:38:36.909766 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:36.892845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54250 (local address 127.0.97.2:39197)
0928 08:38:36.893045 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:36.893052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:36.893074 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:38:36.893283 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:36.893289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:36.893381 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:36.893490 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:36.895120 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.896167 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.898173 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:36.898809 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:36.898940 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:36.903884 (+ 4944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:36.903899 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:36.903915 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:36.903952 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:36.907086 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:36.907890 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:36.907897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:36.907900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:36.908024 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:36.908395 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:36.908398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:36.908399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:36.908764 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:36.908974 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:36.909317 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 08:38:36.909476 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:38:36.910519 11052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:36.998806 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 10929
I20260928 08:38:37.023533 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 11062
Sep 28 08:38:37 dist-test-slave-w6c0 krb5kdc[10689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:38:37.040759 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:38:37.024372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.0.97.3:40447)
0928 08:38:37.024557 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:38:37.024561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:38:37.024578 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:38:37.024775 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:38:37.024779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:38:37.024850 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:38:37.024930 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:38:37.026690 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:37.027470 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:37.029230 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:38:37.029652 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:38:37.029745 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:38:37.034816 (+ 5071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:38:37.034828 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:38:37.034843 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:38:37.034876 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:38:37.037877 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:37.038565 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:37.038579 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:37.038581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:37.038731 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:38:37.039094 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:38:37.039100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:38:37.039103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:38:37.039455 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:38:37.039773 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0928 08:38:37.040087 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 08:38:37.040328 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:38:37.041671 11185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:37.165434 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 11062
I20260928 08:38:37.185257 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 10702
I20260928 08:38:37.186403 10764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:38:37.297894 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 10702
2026-09-28T08:38:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4363 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:38:37 dist-test-slave-w6c0 krb5kdc[11240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:38:37 dist-test-slave-w6c0 krb5kdc[11240](info): set up 2 sockets
Sep 28 08:38:37 dist-test-slave-w6c0 krb5kdc[11240](info): commencing operation
krb5kdc: starting...
W20260928 08:38:38.404119 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:38:38 dist-test-slave-w6c0 krb5kdc[11240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:38:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:38:38Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260928 08:38:38.747859 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task38vn8d/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/postgres -l logfile start
2026-09-28 08:38:41.303 UTC [11270] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 08:38:41.303 UTC [11270] LOG: listening on IPv4 address "127.0.97.20", port 39603
2026-09-28 08:38:41.307 UTC [11270] LOG: listening on Unix socket "/tmp/.s.PGSQL.39603"
2026-09-28 08:38:41.330 UTC [11273] LOG: database system was shut down at 2026-09-28 08:38:39 UTC
2026-09-28 08:38:41.334 UTC [11270] LOG: database system is ready to accept connections
I20260928 08:38:42.315572 388 mini_postgres.cc:96] Postgres bound to 39603
2026-09-28 08:38:42.336 UTC [11281] FATAL: database "slave" does not exist
127.0.97.20:39603 - accepting connections
I20260928 08:38:42.336946 388 mini_ranger.cc:161] Starting Ranger...
I20260928 08:38:42.365561 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 08:38:42.881973 388 mini_ranger.cc:87] Created ranger Postgres database
I20260928 08:38:42.882121 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ranger-admin
2026-09-28 08:38:43,183 [I] DB FLAVOR :POSTGRES
2026-09-28 08:38:43,183 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 08:38:43,183 [I] Checking connection..
2026-09-28 08:38:43,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 08:38:43,472 [I] Checking connection passed.
2026-09-28 08:38:43,472 [I] --------- Verifying version history table ---------
2026-09-28 08:38:43,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:38:43,752 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 08:38:43,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:38:44,022 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 08:38:44,022 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 08:38:44,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 08:38:44.273 UTC [11345] WARNING: there is no transaction in progress
2026-09-28 08:38:44,279 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 08:38:44,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:38:44,551 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 08:38:44,551 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 08:38:44,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 08:38:44,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 08:38:45,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-w6c0.c.gcp-upstream.internal','N') ;"
2026-09-28 08:38:45,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 08:38:45,574 [I] Table x_portal_user does not exist in database ranger
2026-09-28 08:38:45,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 08:38:45,840 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 08:38:45,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 08:38:46,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 08:38:46,307 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 08:38:46,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 08:38:47.249 UTC [11453] WARNING: there is no transaction in progress
2026-09-28 08:38:47.265 UTC [11453] WARNING: there is no transaction in progress
2026-09-28 08:38:47.280 UTC [11453] WARNING: there is no transaction in progress
2026-09-28 08:38:47.295 UTC [11453] WARNING: there is no transaction in progress
2026-09-28 08:38:47.311 UTC [11453] WARNING: there is no transaction in progress
2026-09-28 08:38:47.328 UTC [11453] WARNING: there is no transaction in progress
2026-09-28 08:38:47.382 UTC [11453] WARNING: there is no transaction in progress
2026-09-28 08:38:47.388 UTC [11453] WARNING: there is no transaction in progress
2026-09-28 08:38:47.403 UTC [11453] WARNING: there is no transaction in progress
2026-09-28 08:38:47.412 UTC [11453] WARNING: there is no transaction in progress
2026-09-28 08:38:47.422 UTC [11453] WARNING: there is no transaction in progress
2026-09-28 08:38:47.435 UTC [11453] WARNING: there is no transaction in progress
2026-09-28 08:38:47.445 UTC [11453] WARNING: there is no transaction in progress
2026-09-28 08:38:47.451 UTC [11453] WARNING: there is no transaction in progress
2026-09-28 08:38:47.457 UTC [11453] WARNING: there is no transaction in progress
2026-09-28 08:38:47.463 UTC [11453] WARNING: there is no transaction in progress
2026-09-28 08:38:48,058 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 08:38:48,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 08:38:48,304 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 08:38:48,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 08:38:48,551 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 08:38:48,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 08:38:48,792 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 08:38:48,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 08:38:49,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 08:38:49,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-w6c0.c.gcp-upstream.internal';"
2026-09-28 08:38:49,539 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 08:38:49,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 08:38:49,765 [I] DB_PATCHES have already been applied
I20260928 08:38:49.771073 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task38vn8d/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 08:38:49.771164 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260928 08:38:49.786144 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:49287
Sep 28, 2026 8:38:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:38:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:38:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 8:38:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 8:38:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 8:38:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 8:38:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 08:38:50.905179 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.568s user 0.002s sys 0.039s
Sep 28, 2026 8:39:10 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 8:39:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 8:39:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 8:39:10 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 8:39:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 8:39:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 8:39:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 8:39:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 8:39:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 8:39:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 8:39:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 8:39:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 8:39:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 8:39:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 8:39:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 8:39:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 8:39:10 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 8:39:11 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 08:39:11.731860 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 08:39:11 dist-test-slave-w6c0 krb5kdc[11240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584751, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 08:39:11.912808 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 08:39:11.933755 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 08:39:12.803115 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 08:39:12.803282 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ranger-kms
I20260928 08:39:12.803411 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:39603
2026-09-28 08:39:13,246 [I] DB FLAVOR :POSTGRES
2026-09-28 08:39:13,247 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 08:39:13,247 [I] Checking connection
2026-09-28 08:39:13,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 08:39:13,543 [I] connection success
2026-09-28 08:39:13,544 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 08:39:13,544 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 08:39:13,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 08:39:13,879 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 08:39:13,879 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 08:39:13,880 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 08:39:13,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:39603/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 08:39:14,202 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 08:39:14.711126 388 mini_ranger_kms.cc:310] Created kms service
I20260928 08:39:14.941259 388 mini_ranger_kms.cc:326] Created kudu user
I20260928 08:39:14.991056 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 08:39:14 dist-test-slave-w6c0 krb5kdc[11240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584751, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:39:15.437469 388 mini_ranger_kms.cc:384] Added ranger policy
I20260928 08:39:15.437883 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 08:39:15.437937 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260928 08:39:15.467458 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11633
I20260928 08:39:15.467694 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:55227
Sep 28, 2026 8:39:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:39:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:39:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 8:39:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 8:39:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 8:39:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 08:39:17.937455 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.025s user 0.112s sys 0.352s
I20260928 08:39:17.937740 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 08:39:17.937991 388 mini_ranger_kms.cc:401] 127.0.97.20:55227/kms/v1/keys
Sep 28 08:39:18 dist-test-slave-w6c0 krb5kdc[11240](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790584758, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 08:39:18 dist-test-slave-w6c0 krb5kdc[11240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584758, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 28 08:39:19 dist-test-slave-w6c0 krb5kdc[11240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584758, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 28 08:39:21 dist-test-slave-w6c0 krb5kdc[11240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:39:21.598407 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45801
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45801
--ranger_config_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55227/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:39:21.979555 11684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:39:21.979871 11684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:39:21.980042 11684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:39:21.988080 11684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:39:21.988199 11684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:39:21.988293 11684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:39:21.988407 11684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:39:21.988476 11684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:39:21.998982 11684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:55227/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45801
--ranger_config_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:39:22.000402 11684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:39:22.001755 11684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:39:22.014626 11689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:39:22.014705 11690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:39:22.015308 11692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:39:22.015723 11684 server_base.cc:1034] running on GCE node
I20260928 08:39:22.016557 11684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:39:22.018141 11684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:39:22.019362 11684 hybrid_clock.cc:648] HybridClock initialized: now 1790584762019316 us; error 64 us; skew 500 ppm
I20260928 08:39:22.019730 11684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:39:22 dist-test-slave-w6c0 krb5kdc[11240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:39:22.028980 11684 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:39:22.030944 11684 webserver.cc:460] Webserver started at http://127.0.97.62:36087/ using document root <none> and password file <none>
I20260928 08:39:22.031683 11684 fs_manager.cc:359] Metadata directory not provided
I20260928 08:39:22.031828 11684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:39:22.032145 11684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:39:22 dist-test-slave-w6c0 krb5kdc[11240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:39:22.094791 11684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "d7156a0280ed45c59ffc4c09e36dc707"
format_stamp: "Formatted at 2026-09-28 08:39:22 on dist-test-slave-w6c0"
server_key: "f9d205530f425ba75d6b691cc9584f5e"
server_key_iv: "ebb35b9698ccf6c65177e2af01092dc0"
server_key_version: "kuduclusterkey@0"
I20260928 08:39:22.095955 11684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "d7156a0280ed45c59ffc4c09e36dc707"
format_stamp: "Formatted at 2026-09-28 08:39:22 on dist-test-slave-w6c0"
server_key: "f9d205530f425ba75d6b691cc9584f5e"
server_key_iv: "ebb35b9698ccf6c65177e2af01092dc0"
server_key_version: "kuduclusterkey@0"
I20260928 08:39:22.103421 11684 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260928 08:39:22.123701 11699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:39:22.125263 11684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:39:22.125425 11684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "d7156a0280ed45c59ffc4c09e36dc707"
format_stamp: "Formatted at 2026-09-28 08:39:22 on dist-test-slave-w6c0"
server_key: "f9d205530f425ba75d6b691cc9584f5e"
server_key_iv: "ebb35b9698ccf6c65177e2af01092dc0"
server_key_version: "kuduclusterkey@0"
I20260928 08:39:22.125619 11684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:39:22.146421 11684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:39:22.152376 11684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:39:22.152756 11684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:39:22.173897 11684 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45801
I20260928 08:39:22.173914 11750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45801 every 8 connection(s)
I20260928 08:39:22.175419 11684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:39:22.178197 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 11684
I20260928 08:39:22.178399 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:39:22.179073 388 external_mini_cluster.cc:1428] Setting key d3f82f792568718d77414336e3726574
I20260928 08:39:22.179701 11751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:39:22.194741 11751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707: Bootstrap starting.
Sep 28 08:39:22 dist-test-slave-w6c0 krb5kdc[11240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:39:22.199096 11751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707: Neither blocks nor log segments found. Creating new log.
I20260928 08:39:22.200249 11751 log.cc:824] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707: Log is configured to *not* fsync() on all Append() calls
I20260928 08:39:22.203758 11751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707: No bootstrap required, opened a new log
I20260928 08:39:22.204505 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:22.181502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35858 (local address 127.0.97.62:45801)
0928 08:39:22.182342 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0928 08:39:22.182358 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:39:22.182414 (+ 56us) server_negotiation.cc:407] Connection header received
0928 08:39:22.184729 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:39:22.184761 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:39:22.185331 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:39:22.185835 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:39:22.187324 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:22.189360 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:22.190762 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:22.191925 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:22.192130 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:22.196456 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:39:22.196470 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:39:22.196485 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:39:22.196533 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:39:22.200084 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:39:22.201049 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:39:22.201065 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:39:22.201073 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:39:22.201290 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:39:22.201726 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:39:22.201733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:39:22.201736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:39:22.202259 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:39:22.202482 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 08:39:22.202989 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 08:39:22.203355 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":364,"threads_started":1}
I20260928 08:39:22.212637 11751 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7156a0280ed45c59ffc4c09e36dc707" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45801 } }
I20260928 08:39:22.213461 11751 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:39:22.213569 11751 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7156a0280ed45c59ffc4c09e36dc707, State: Initialized, Role: FOLLOWER
I20260928 08:39:22.214178 11751 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7156a0280ed45c59ffc4c09e36dc707" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45801 } }
I20260928 08:39:22.214416 11751 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:39:22.214602 11751 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:39:22.214788 11751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:39:22.217293 11751 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7156a0280ed45c59ffc4c09e36dc707" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45801 } }
I20260928 08:39:22.217929 11751 leader_election.cc:302] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7156a0280ed45c59ffc4c09e36dc707; no voters:
I20260928 08:39:22.218464 11751 leader_election.cc:288] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:39:22.218667 11756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:39:22.219350 11756 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707 [term 1 LEADER]: Becoming Leader. State: Replica: d7156a0280ed45c59ffc4c09e36dc707, State: Running, Role: LEADER
I20260928 08:39:22.219928 11756 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7156a0280ed45c59ffc4c09e36dc707" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45801 } }
I20260928 08:39:22.220717 11751 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:39:22.222714 11757 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7156a0280ed45c59ffc4c09e36dc707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7156a0280ed45c59ffc4c09e36dc707" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45801 } } }
I20260928 08:39:22.223142 11757 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707 [sys.catalog]: This master's current role is: LEADER
I20260928 08:39:22.222774 11758 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7156a0280ed45c59ffc4c09e36dc707. Latest consensus state: current_term: 1 leader_uuid: "d7156a0280ed45c59ffc4c09e36dc707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7156a0280ed45c59ffc4c09e36dc707" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45801 } } }
I20260928 08:39:22.223419 11758 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707 [sys.catalog]: This master's current role is: LEADER
I20260928 08:39:22.225216 11764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:39:22.230434 11764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:39:22.231213 11751 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 08:39:22.248279 11764 catalog_manager.cc:1347] Generated new cluster ID: d1d41af707fe49c0ae316e217b8ab9f0
I20260928 08:39:22.248452 11764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:39:22.272729 11764 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:39:22.274204 11764 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:39:22.283170 11764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707: Generated new TSK 0
I20260928 08:39:22.284327 11764 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:39:23.026839 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45801
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:39:23 dist-test-slave-w6c0 krb5kdc[11240](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790584763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 08:39:23.454486 11796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:39:23.454900 11796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:39:23.455114 11796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:39:23.466720 11796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:39:23.466919 11796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:39:23.467149 11796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:39:23.483062 11796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:55227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:39:23.484898 11796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:39:23.486615 11796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:39:23.525364 11806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:39:23.528128 11807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:39:23.529278 11809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:39:23.549779 11796 server_base.cc:1034] running on GCE node
I20260928 08:39:23.550701 11796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:39:23.558990 11796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:39:23.562425 11796 hybrid_clock.cc:648] HybridClock initialized: now 1790584763562296 us; error 129 us; skew 500 ppm
I20260928 08:39:23.562883 11796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:39:23 dist-test-slave-w6c0 krb5kdc[11240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:39:23.567495 11796 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:39:23.579365 11796 webserver.cc:460] Webserver started at http://127.0.97.1:45105/ using document root <none> and password file <none>
I20260928 08:39:23.580160 11796 fs_manager.cc:359] Metadata directory not provided
I20260928 08:39:23.580324 11796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:39:23.580677 11796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:39:23 dist-test-slave-w6c0 krb5kdc[11240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:39:23.600409 11796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "f3fd0679742f47a883959801ac42f637"
format_stamp: "Formatted at 2026-09-28 08:39:23 on dist-test-slave-w6c0"
server_key: "b9c43135890ceaee0216c6c9bda798a8"
server_key_iv: "b9f99d655bac758d9b9a6e87a7891ffb"
server_key_version: "kuduclusterkey@0"
I20260928 08:39:23.601771 11796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f3fd0679742f47a883959801ac42f637"
format_stamp: "Formatted at 2026-09-28 08:39:23 on dist-test-slave-w6c0"
server_key: "b9c43135890ceaee0216c6c9bda798a8"
server_key_iv: "b9f99d655bac758d9b9a6e87a7891ffb"
server_key_version: "kuduclusterkey@0"
I20260928 08:39:23.611929 11796 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.005s
I20260928 08:39:23.633333 11816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:39:23.635437 11796 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260928 08:39:23.635708 11796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "f3fd0679742f47a883959801ac42f637"
format_stamp: "Formatted at 2026-09-28 08:39:23 on dist-test-slave-w6c0"
server_key: "b9c43135890ceaee0216c6c9bda798a8"
server_key_iv: "b9f99d655bac758d9b9a6e87a7891ffb"
server_key_version: "kuduclusterkey@0"
I20260928 08:39:23.636112 11796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:39:23.674469 11796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:39:23.680897 11796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:39:23.681313 11796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:39:23.682617 11796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:39:23.684487 11796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:39:23.684628 11796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 08:39:23.684842 11796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:39:23.684988 11796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:39:23.747488 11796 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38153
I20260928 08:39:23.747529 11928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38153 every 8 connection(s)
I20260928 08:39:23.749233 11796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:39:23.751513 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 11796
I20260928 08:39:23.751721 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:39:23.752553 388 external_mini_cluster.cc:1428] Setting key 93ee1b1fa326c0c4283cece3978db282
Sep 28 08:39:23 dist-test-slave-w6c0 krb5kdc[11240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:39:23.808406 11933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:23.766484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46161 (local address 127.0.97.62:45801)
0928 08:39:23.784419 (+ 17935us) server_negotiation.cc:206] Beginning negotiation
0928 08:39:23.784426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:39:23.784451 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:39:23.784526 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:39:23.784532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:39:23.784632 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:39:23.784762 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:39:23.788267 (+ 3505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:23.789433 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:23.792246 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:23.792878 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:23.792985 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:23.799114 (+ 6129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:39:23.799121 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:39:23.799126 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:39:23.799166 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:39:23.803480 (+ 4314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:39:23.804482 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:39:23.804487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:39:23.804490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:39:23.804652 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:39:23.805390 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:39:23.805395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:39:23.805398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:39:23.806125 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:39:23.806260 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 08:39:23.807492 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0928 08:39:23.807729 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17766,"thread_start_us":16443,"threads_started":1}
I20260928 08:39:23.811112 11932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:23.765828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45801 (local address 127.0.97.1:46161)
0928 08:39:23.767713 (+ 1885us) negotiation.cc:107] Waiting for socket to connect
0928 08:39:23.767772 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 08:39:23.777365 (+ 9593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:39:23.786308 (+ 8943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:39:23.786324 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:39:23.787085 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:39:23.788016 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:39:23.788052 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:39:23.789663 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:39:23.789668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:39:23.792059 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:39:23.792076 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:39:23.793344 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:39:23.793352 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:39:23.793649 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:23.794757 (+ 1108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:39:23.794790 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:39:23.798813 (+ 4023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:39:23.803674 (+ 4861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:39:23.803685 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:39:23.803707 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:39:23.804333 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:39:23.804973 (+ 640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:39:23.804977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:39:23.804980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:39:23.805233 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:39:23.806371 (+ 1138us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:39:23.806380 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:39:23.807001 (+ 621us) client_negotiation.cc:769] Sending connection context
0928 08:39:23.808892 (+ 1891us) client_negotiation.cc:240] Negotiation successful
0928 08:39:23.809634 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1560,"thread_start_us":1289,"threads_started":1}
Sep 28 08:39:23 dist-test-slave-w6c0 krb5kdc[11240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:39:23.814167 11929 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45801
I20260928 08:39:23.814545 11929 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:39:23.815495 11929 heartbeater.cc:511] Master 127.0.97.62:45801 requested a full tablet report, sending...
I20260928 08:39:23.818471 11716 ts_manager.cc:194] Registered new tserver with Master: f3fd0679742f47a883959801ac42f637 (127.0.97.1:38153)
I20260928 08:39:23.820542 11716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
W20260928 08:39:23.859236 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.2: real 0.104s user 0.000s sys 0.017s
I20260928 08:39:23.860181 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45801
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:39:24 dist-test-slave-w6c0 krb5kdc[11240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260928 08:39:24.392374 11935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:39:24.392897 11935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:39:24.393101 11935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:39:24.405074 11935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:39:24.405215 11935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:39:24.405419 11935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:39:24.421298 11935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:55227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:39:24.423400 11935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:39:24.425182 11935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:39:24.455070 11943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:39:24.456148 11942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:39:24.462311 11935 server_base.cc:1034] running on GCE node
W20260928 08:39:24.462782 11946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:39:24.463841 11935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:39:24.465142 11935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:39:24.466439 11935 hybrid_clock.cc:648] HybridClock initialized: now 1790584764466371 us; error 55 us; skew 500 ppm
I20260928 08:39:24.466879 11935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:39:24 dist-test-slave-w6c0 krb5kdc[11240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:39:24.471004 11935 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:39:24.479230 11935 webserver.cc:460] Webserver started at http://127.0.97.2:45239/ using document root <none> and password file <none>
I20260928 08:39:24.480165 11935 fs_manager.cc:359] Metadata directory not provided
I20260928 08:39:24.480329 11935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:39:24.480790 11935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:39:24 dist-test-slave-w6c0 krb5kdc[11240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:39:24.512305 11935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "b7e7bc01c5f54731ab1fff9a963f6e08"
format_stamp: "Formatted at 2026-09-28 08:39:24 on dist-test-slave-w6c0"
server_key: "cf5ae5f5af5239098a7496b3788055f2"
server_key_iv: "2af5831d26cc167a923a29ae854e2dfe"
server_key_version: "kuduclusterkey@0"
I20260928 08:39:24.513612 11935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b7e7bc01c5f54731ab1fff9a963f6e08"
format_stamp: "Formatted at 2026-09-28 08:39:24 on dist-test-slave-w6c0"
server_key: "cf5ae5f5af5239098a7496b3788055f2"
server_key_iv: "2af5831d26cc167a923a29ae854e2dfe"
server_key_version: "kuduclusterkey@0"
I20260928 08:39:24.520509 11935 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 08:39:24.538795 11959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:39:24.540382 11935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:39:24.540584 11935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "b7e7bc01c5f54731ab1fff9a963f6e08"
format_stamp: "Formatted at 2026-09-28 08:39:24 on dist-test-slave-w6c0"
server_key: "cf5ae5f5af5239098a7496b3788055f2"
server_key_iv: "2af5831d26cc167a923a29ae854e2dfe"
server_key_version: "kuduclusterkey@0"
I20260928 08:39:24.540823 11935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:39:24.554466 11935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:39:24.558694 11935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:39:24.558993 11935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:39:24.559921 11935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:39:24.561354 11935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:39:24.561460 11935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:39:24.561606 11935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:39:24.561676 11935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:39:24.603626 11935 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45885
I20260928 08:39:24.603621 12071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45885 every 8 connection(s)
I20260928 08:39:24.605248 11935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:39:24.615195 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 11935
I20260928 08:39:24.615366 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:39:24.615924 388 external_mini_cluster.cc:1428] Setting key e570cfdf85781323a05ebc9952aa7fd8
Sep 28 08:39:24 dist-test-slave-w6c0 krb5kdc[11240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:39:24.627385 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:24.608302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38699 (local address 127.0.97.62:45801)
0928 08:39:24.608791 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0928 08:39:24.608797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:39:24.609797 (+ 1000us) server_negotiation.cc:407] Connection header received
0928 08:39:24.611626 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:39:24.611633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:39:24.611724 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:39:24.611853 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:39:24.613138 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:24.614275 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:24.616774 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:24.617312 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:24.617423 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:24.621113 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:39:24.621118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:39:24.621121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:39:24.621150 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:39:24.624284 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:39:24.624981 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:39:24.624987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:39:24.624990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:39:24.625107 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:39:24.625536 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:39:24.625541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:39:24.625544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:39:24.625810 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:39:24.625992 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 08:39:24.626776 (+ 784us) server_negotiation.cc:299] Negotiation successful
0928 08:39:24.627026 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":241,"threads_started":1}
I20260928 08:39:24.628577 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:24.608805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45801 (local address 127.0.97.2:38699)
0928 08:39:24.609626 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0928 08:39:24.609676 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:39:24.611359 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:39:24.612030 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:39:24.612037 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:39:24.612459 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:39:24.612975 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:39:24.612991 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:39:24.614426 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:39:24.614430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:39:24.616390 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:39:24.616416 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:39:24.617773 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:39:24.617782 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:39:24.618060 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:24.618780 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:39:24.618801 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:39:24.620824 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:39:24.624422 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:39:24.624433 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:39:24.624449 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:39:24.624855 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:39:24.625215 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:39:24.625220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:39:24.625224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:39:24.625422 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:39:24.625943 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:39:24.625951 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:39:24.626360 (+ 409us) client_negotiation.cc:769] Sending connection context
0928 08:39:24.626683 (+ 323us) client_negotiation.cc:240] Negotiation successful
0928 08:39:24.627176 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":365,"threads_started":1}
I20260928 08:39:24.630013 12072 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45801
I20260928 08:39:24.630309 12072 heartbeater.cc:464] Registering TS with master...
I20260928 08:39:24.631186 12072 heartbeater.cc:511] Master 127.0.97.62:45801 requested a full tablet report, sending...
I20260928 08:39:24.632812 11716 ts_manager.cc:194] Registered new tserver with Master: b7e7bc01c5f54731ab1fff9a963f6e08 (127.0.97.2:45885)
I20260928 08:39:24.634042 11716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38699
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:39:24.673565 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45801
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:39:24.825294 11929 heartbeater.cc:503] Master 127.0.97.62:45801 was elected leader, sending a full tablet report...
W20260928 08:39:25.005894 12078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:39:25.006196 12078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:39:25.006373 12078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:39:25.013487 12078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:39:25.013584 12078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:39:25.013748 12078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:39:25.023594 12078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:55227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:39:25.024874 12078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:39:25.026214 12078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:39:25.039852 12083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:39:25.039906 12084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:39:25.040038 12086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:39:25.040693 12078 server_base.cc:1034] running on GCE node
I20260928 08:39:25.041221 12078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:39:25.042150 12078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:39:25.043363 12078 hybrid_clock.cc:648] HybridClock initialized: now 1790584765043320 us; error 51 us; skew 500 ppm
I20260928 08:39:25.043697 12078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:39:25 dist-test-slave-w6c0 krb5kdc[11240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:39:25.047111 12078 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:39:25.048563 12078 webserver.cc:460] Webserver started at http://127.0.97.3:40903/ using document root <none> and password file <none>
I20260928 08:39:25.049219 12078 fs_manager.cc:359] Metadata directory not provided
I20260928 08:39:25.049362 12078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:39:25.049722 12078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:39:25 dist-test-slave-w6c0 krb5kdc[11240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:39:25.067142 12078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "4a78e5b9682d4419b4c5b2b716efaddc"
format_stamp: "Formatted at 2026-09-28 08:39:25 on dist-test-slave-w6c0"
server_key: "3ab9a7b442fb03f5a175b9ca1800e868"
server_key_iv: "5eae88ba6fe022e6e74c4a861a90fe8d"
server_key_version: "kuduclusterkey@0"
I20260928 08:39:25.068153 12078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4a78e5b9682d4419b4c5b2b716efaddc"
format_stamp: "Formatted at 2026-09-28 08:39:25 on dist-test-slave-w6c0"
server_key: "3ab9a7b442fb03f5a175b9ca1800e868"
server_key_iv: "5eae88ba6fe022e6e74c4a861a90fe8d"
server_key_version: "kuduclusterkey@0"
I20260928 08:39:25.073625 12078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:39:25.090184 12093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:39:25.091835 12078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:39:25.092032 12078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "4a78e5b9682d4419b4c5b2b716efaddc"
format_stamp: "Formatted at 2026-09-28 08:39:25 on dist-test-slave-w6c0"
server_key: "3ab9a7b442fb03f5a175b9ca1800e868"
server_key_iv: "5eae88ba6fe022e6e74c4a861a90fe8d"
server_key_version: "kuduclusterkey@0"
I20260928 08:39:25.092254 12078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:39:25.108834 12078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:39:25.112798 12078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:39:25.113091 12078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:39:25.113982 12078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:39:25.115365 12078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:39:25.115465 12078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:39:25.115614 12078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:39:25.115684 12078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:39:25.149735 12078 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38807
I20260928 08:39:25.149766 12205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38807 every 8 connection(s)
I20260928 08:39:25.151082 12078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:39:25.158444 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 12078
I20260928 08:39:25.158665 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:39:25.159204 388 external_mini_cluster.cc:1428] Setting key 10938d9e68d129df8b5f93e0322ac242
Sep 28 08:39:25 dist-test-slave-w6c0 krb5kdc[11240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:39:25.170162 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:25.153444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34627 (local address 127.0.97.62:45801)
0928 08:39:25.153902 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0928 08:39:25.153908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:39:25.155333 (+ 1425us) server_negotiation.cc:407] Connection header received
0928 08:39:25.156970 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:39:25.156975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:39:25.157051 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:39:25.157146 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:39:25.158462 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.159579 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.161516 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.162055 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.162194 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:25.165230 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:39:25.165235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:39:25.165238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:39:25.165268 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:39:25.167705 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:39:25.168346 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:39:25.168350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:39:25.168352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:39:25.168461 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:39:25.168820 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:39:25.168823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:39:25.168825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:39:25.169075 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:39:25.169179 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:39:25.169765 (+ 586us) server_negotiation.cc:299] Negotiation successful
0928 08:39:25.169913 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":201,"threads_started":1}
I20260928 08:39:25.171067 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:25.153963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45801 (local address 127.0.97.3:34627)
0928 08:39:25.155175 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0928 08:39:25.155216 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:39:25.156740 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:39:25.157342 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:39:25.157349 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:39:25.157760 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:39:25.158296 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:39:25.158312 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.159751 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.159756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:39:25.161382 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:39:25.161396 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.162268 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.162272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:39:25.162463 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:25.163121 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:39:25.163143 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:39:25.165039 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:39:25.167820 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:39:25.167826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:39:25.167836 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:39:25.168237 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:39:25.168547 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:39:25.168550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:39:25.168553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:39:25.168731 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:39:25.169179 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:39:25.169183 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:39:25.169429 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 08:39:25.169703 (+ 274us) client_negotiation.cc:240] Negotiation successful
0928 08:39:25.170025 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"thread_start_us":292,"threads_started":1}
I20260928 08:39:25.172402 12206 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45801
I20260928 08:39:25.172680 12206 heartbeater.cc:464] Registering TS with master...
I20260928 08:39:25.173440 12206 heartbeater.cc:511] Master 127.0.97.62:45801 requested a full tablet report, sending...
I20260928 08:39:25.174963 11716 ts_manager.cc:194] Registered new tserver with Master: 4a78e5b9682d4419b4c5b2b716efaddc (127.0.97.3:38807)
I20260928 08:39:25.176038 11716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34627
I20260928 08:39:25.186586 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:39:25.205024 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:25.190875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.0.97.62:45801)
0928 08:39:25.191070 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 08:39:25.191078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:39:25.191526 (+ 448us) server_negotiation.cc:407] Connection header received
0928 08:39:25.191649 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:39:25.191653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:39:25.191731 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:39:25.191808 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:39:25.193165 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.194409 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.196147 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.196960 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.197103 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:25.200218 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:39:25.200224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:39:25.200229 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:39:25.200271 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:39:25.202481 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:39:25.203150 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:39:25.203158 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:39:25.203162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:39:25.203291 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:39:25.203652 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:39:25.203659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:39:25.203662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:39:25.203935 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:39:25.204179 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 08:39:25.204493 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 08:39:25.204719 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:39:25.209316 11716 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:39:25.216454 11716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:39:25.238843 12220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:25.228496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38807 (local address 127.0.0.1:56466)
0928 08:39:25.230168 (+ 1672us) negotiation.cc:107] Waiting for socket to connect
0928 08:39:25.230185 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:39:25.230290 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:39:25.231376 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:39:25.231381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:39:25.231403 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:39:25.231806 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:39:25.231818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.234312 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.234317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:39:25.235885 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:39:25.235896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.237736 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.237742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:39:25.237945 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:25.238074 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 08:39:25.238180 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 08:39:25.238367 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1563,"mutex_wait_us":53,"thread_start_us":1427,"threads_started":1}
I20260928 08:39:25.238909 12222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:25.229265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56466 (local address 127.0.97.3:38807)
0928 08:39:25.230815 (+ 1550us) server_negotiation.cc:206] Beginning negotiation
0928 08:39:25.230825 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:39:25.230857 (+ 32us) server_negotiation.cc:407] Connection header received
0928 08:39:25.230951 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:39:25.230959 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:39:25.231162 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:39:25.231980 (+ 818us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:39:25.232221 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.234160 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.236088 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.237512 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.238009 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:25.238259 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 08:39:25.238320 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 08:39:25.238471 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1398,"thread_start_us":2536,"threads_started":1}
I20260928 08:39:25.239217 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:25.228303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45885 (local address 127.0.0.1:44142)
0928 08:39:25.229385 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0928 08:39:25.229441 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 08:39:25.229617 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:39:25.230481 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:39:25.230492 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:39:25.230550 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:39:25.230953 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:39:25.230968 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.233602 (+ 2634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.233608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:39:25.236119 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:39:25.236130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.238399 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.238404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:39:25.238612 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:25.238641 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 08:39:25.238738 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 08:39:25.238880 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":131,"thread_start_us":998,"threads_started":1}
I20260928 08:39:25.239830 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:25.229146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44142 (local address 127.0.97.2:45885)
0928 08:39:25.229781 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0928 08:39:25.229792 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:39:25.229822 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:39:25.229923 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:39:25.229933 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:39:25.230197 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:39:25.231169 (+ 972us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:39:25.231403 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.232991 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.236261 (+ 3270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.237923 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.238072 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:25.238267 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 08:39:25.239324 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0928 08:39:25.239490 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":1539,"threads_started":1}
I20260928 08:39:25.240587 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:25.231003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38153 (local address 127.0.0.1:37540)
0928 08:39:25.231496 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 08:39:25.231515 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:39:25.231630 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:39:25.232629 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:39:25.232633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:39:25.232657 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:39:25.233010 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:39:25.233023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.235992 (+ 2969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.236000 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:39:25.237981 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:39:25.237992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.239930 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.239935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:39:25.240039 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:25.240060 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 08:39:25.240175 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 08:39:25.240302 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":964,"threads_started":1}
I20260928 08:39:25.241449 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:25.231574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.0.97.1:38153)
0928 08:39:25.232175 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0928 08:39:25.232182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:39:25.232217 (+ 35us) server_negotiation.cc:407] Connection header received
0928 08:39:25.232310 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:39:25.232315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:39:25.232453 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:39:25.232629 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:39:25.234358 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.235845 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.238159 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.239514 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.239639 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:25.239866 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 08:39:25.240927 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0928 08:39:25.241096 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":487,"threads_started":1}
I20260928 08:39:25.243407 12007 tablet_service.cc:1467] Processing CreateTablet for tablet 78e8c905597a4874ae12915ed74d1ce7 (DEFAULT_TABLE table=test-table [id=3d7b3e9dc86d488994e355eebd26e647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:39:25.243407 12141 tablet_service.cc:1467] Processing CreateTablet for tablet 78e8c905597a4874ae12915ed74d1ce7 (DEFAULT_TABLE table=test-table [id=3d7b3e9dc86d488994e355eebd26e647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:39:25.244827 12007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e8c905597a4874ae12915ed74d1ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:39:25.244828 12141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e8c905597a4874ae12915ed74d1ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:39:25.244542 11864 tablet_service.cc:1467] Processing CreateTablet for tablet 78e8c905597a4874ae12915ed74d1ce7 (DEFAULT_TABLE table=test-table [id=3d7b3e9dc86d488994e355eebd26e647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:39:25.245666 11864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e8c905597a4874ae12915ed74d1ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:39:25.256912 12225 tablet_bootstrap.cc:492] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637: Bootstrap starting.
I20260928 08:39:25.259262 12226 tablet_bootstrap.cc:492] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc: Bootstrap starting.
I20260928 08:39:25.260838 12227 tablet_bootstrap.cc:492] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08: Bootstrap starting.
I20260928 08:39:25.261065 12225 tablet_bootstrap.cc:654] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637: Neither blocks nor log segments found. Creating new log.
I20260928 08:39:25.262807 12225 log.cc:824] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637: Log is configured to *not* fsync() on all Append() calls
I20260928 08:39:25.264647 12227 tablet_bootstrap.cc:654] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08: Neither blocks nor log segments found. Creating new log.
I20260928 08:39:25.265357 12226 tablet_bootstrap.cc:654] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc: Neither blocks nor log segments found. Creating new log.
I20260928 08:39:25.265802 12227 log.cc:824] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08: Log is configured to *not* fsync() on all Append() calls
I20260928 08:39:25.266002 12225 tablet_bootstrap.cc:492] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637: No bootstrap required, opened a new log
I20260928 08:39:25.266259 12225 ts_tablet_manager.cc:1397] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 08:39:25.266348 12226 log.cc:824] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc: Log is configured to *not* fsync() on all Append() calls
I20260928 08:39:25.269119 12227 tablet_bootstrap.cc:492] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08: No bootstrap required, opened a new log
I20260928 08:39:25.269375 12227 ts_tablet_manager.cc:1397] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260928 08:39:25.269872 12226 tablet_bootstrap.cc:492] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc: No bootstrap required, opened a new log
I20260928 08:39:25.270114 12226 ts_tablet_manager.cc:1397] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260928 08:39:25.276546 12227 raft_consensus.cc:348] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e7bc01c5f54731ab1fff9a963f6e08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45885 } } peers { permanent_uuid: "4a78e5b9682d4419b4c5b2b716efaddc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38807 } } peers { permanent_uuid: "f3fd0679742f47a883959801ac42f637" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38153 } }
I20260928 08:39:25.276619 12225 raft_consensus.cc:348] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e7bc01c5f54731ab1fff9a963f6e08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45885 } } peers { permanent_uuid: "4a78e5b9682d4419b4c5b2b716efaddc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38807 } } peers { permanent_uuid: "f3fd0679742f47a883959801ac42f637" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38153 } }
I20260928 08:39:25.276916 12227 raft_consensus.cc:374] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:39:25.277027 12225 raft_consensus.cc:374] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:39:25.277040 12227 raft_consensus.cc:729] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7e7bc01c5f54731ab1fff9a963f6e08, State: Initialized, Role: FOLLOWER
I20260928 08:39:25.277129 12225 raft_consensus.cc:729] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3fd0679742f47a883959801ac42f637, State: Initialized, Role: FOLLOWER
I20260928 08:39:25.277613 12227 consensus_queue.cc:260] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e7bc01c5f54731ab1fff9a963f6e08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45885 } } peers { permanent_uuid: "4a78e5b9682d4419b4c5b2b716efaddc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38807 } } peers { permanent_uuid: "f3fd0679742f47a883959801ac42f637" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38153 } }
I20260928 08:39:25.277726 12225 consensus_queue.cc:260] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e7bc01c5f54731ab1fff9a963f6e08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45885 } } peers { permanent_uuid: "4a78e5b9682d4419b4c5b2b716efaddc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38807 } } peers { permanent_uuid: "f3fd0679742f47a883959801ac42f637" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38153 } }
I20260928 08:39:25.279506 12227 ts_tablet_manager.cc:1428] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260928 08:39:25.279958 12072 heartbeater.cc:503] Master 127.0.97.62:45801 was elected leader, sending a full tablet report...
I20260928 08:39:25.280776 12226 raft_consensus.cc:348] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e7bc01c5f54731ab1fff9a963f6e08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45885 } } peers { permanent_uuid: "4a78e5b9682d4419b4c5b2b716efaddc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38807 } } peers { permanent_uuid: "f3fd0679742f47a883959801ac42f637" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38153 } }
I20260928 08:39:25.281194 12226 raft_consensus.cc:374] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:39:25.281291 12226 raft_consensus.cc:729] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a78e5b9682d4419b4c5b2b716efaddc, State: Initialized, Role: FOLLOWER
I20260928 08:39:25.281926 12226 consensus_queue.cc:260] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e7bc01c5f54731ab1fff9a963f6e08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45885 } } peers { permanent_uuid: "4a78e5b9682d4419b4c5b2b716efaddc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38807 } } peers { permanent_uuid: "f3fd0679742f47a883959801ac42f637" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38153 } }
I20260928 08:39:25.282123 12225 ts_tablet_manager.cc:1428] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20260928 08:39:25.282974 12206 heartbeater.cc:503] Master 127.0.97.62:45801 was elected leader, sending a full tablet report...
I20260928 08:39:25.283574 12226 ts_tablet_manager.cc:1428] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260928 08:39:25.284926 11930 tablet.cc:2369] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:39:25.296912 12233 raft_consensus.cc:482] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:39:25.297168 12233 raft_consensus.cc:504] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e7bc01c5f54731ab1fff9a963f6e08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45885 } } peers { permanent_uuid: "4a78e5b9682d4419b4c5b2b716efaddc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38807 } } peers { permanent_uuid: "f3fd0679742f47a883959801ac42f637" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38153 } }
I20260928 08:39:25.298868 12233 leader_election.cc:288] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7e7bc01c5f54731ab1fff9a963f6e08 (127.0.97.2:45885), f3fd0679742f47a883959801ac42f637 (127.0.97.1:38153)
I20260928 08:39:25.304864 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:25.299300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34549 (local address 127.0.97.2:45885)
0928 08:39:25.299514 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 08:39:25.299522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:39:25.299543 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:39:25.299625 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:39:25.299630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:39:25.299699 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:39:25.299820 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:39:25.300491 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.301468 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.303197 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.303968 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.304113 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:25.304212 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 08:39:25.304481 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 08:39:25.304618 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:39:25.304880 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:25.299125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45885 (local address 127.0.97.3:34549)
0928 08:39:25.299300 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 08:39:25.299321 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:39:25.299445 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:39:25.300045 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:39:25.300050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:39:25.300085 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:39:25.300298 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:39:25.300309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.301599 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.301608 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:39:25.303034 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:39:25.303046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.304128 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.304134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:39:25.304318 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:25.304342 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 08:39:25.304442 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 08:39:25.304594 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":4}
I20260928 08:39:25.305712 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:25.299827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41281 (local address 127.0.97.1:38153)
0928 08:39:25.300038 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 08:39:25.300052 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:39:25.300298 (+ 246us) server_negotiation.cc:407] Connection header received
0928 08:39:25.300441 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:39:25.300446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:39:25.300520 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:39:25.300606 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:39:25.301158 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.301945 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.303661 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.304409 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.304563 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:25.304673 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 08:39:25.304931 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 08:39:25.305055 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:39:25.305824 12234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:25.299530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38153 (local address 127.0.97.3:41281)
0928 08:39:25.300188 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0928 08:39:25.300205 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:39:25.300321 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:39:25.300652 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:39:25.300656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:39:25.300682 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:39:25.301003 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:39:25.301013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.302102 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.302106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:39:25.303497 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:39:25.303509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.304609 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.304615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:39:25.304774 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:25.304796 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 08:39:25.304905 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 08:39:25.305019 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":332,"threads_started":1}
I20260928 08:39:25.306126 12027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e8c905597a4874ae12915ed74d1ce7" candidate_uuid: "4a78e5b9682d4419b4c5b2b716efaddc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7e7bc01c5f54731ab1fff9a963f6e08" is_pre_election: true
I20260928 08:39:25.306674 12027 raft_consensus.cc:2454] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a78e5b9682d4419b4c5b2b716efaddc in term 0.
I20260928 08:39:25.306859 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e8c905597a4874ae12915ed74d1ce7" candidate_uuid: "4a78e5b9682d4419b4c5b2b716efaddc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3fd0679742f47a883959801ac42f637" is_pre_election: true
I20260928 08:39:25.307312 12096 leader_election.cc:302] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a78e5b9682d4419b4c5b2b716efaddc, b7e7bc01c5f54731ab1fff9a963f6e08; no voters:
I20260928 08:39:25.307324 11884 raft_consensus.cc:2454] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a78e5b9682d4419b4c5b2b716efaddc in term 0.
I20260928 08:39:25.307653 12233 raft_consensus.cc:2781] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:39:25.307792 12233 raft_consensus.cc:482] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:39:25.307893 12233 raft_consensus.cc:3037] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:39:25.310145 12233 raft_consensus.cc:504] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e7bc01c5f54731ab1fff9a963f6e08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45885 } } peers { permanent_uuid: "4a78e5b9682d4419b4c5b2b716efaddc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38807 } } peers { permanent_uuid: "f3fd0679742f47a883959801ac42f637" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38153 } }
I20260928 08:39:25.310742 12233 leader_election.cc:288] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [CANDIDATE]: Term 1 election: Requested vote from peers b7e7bc01c5f54731ab1fff9a963f6e08 (127.0.97.2:45885), f3fd0679742f47a883959801ac42f637 (127.0.97.1:38153)
I20260928 08:39:25.311211 12027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e8c905597a4874ae12915ed74d1ce7" candidate_uuid: "4a78e5b9682d4419b4c5b2b716efaddc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7e7bc01c5f54731ab1fff9a963f6e08"
I20260928 08:39:25.311295 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e8c905597a4874ae12915ed74d1ce7" candidate_uuid: "4a78e5b9682d4419b4c5b2b716efaddc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3fd0679742f47a883959801ac42f637"
I20260928 08:39:25.311408 12027 raft_consensus.cc:3037] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:39:25.311520 11884 raft_consensus.cc:3037] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:39:25.313632 12027 raft_consensus.cc:2454] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a78e5b9682d4419b4c5b2b716efaddc in term 1.
I20260928 08:39:25.313659 11884 raft_consensus.cc:2454] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a78e5b9682d4419b4c5b2b716efaddc in term 1.
I20260928 08:39:25.313995 12096 leader_election.cc:302] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a78e5b9682d4419b4c5b2b716efaddc, b7e7bc01c5f54731ab1fff9a963f6e08; no voters:
I20260928 08:39:25.314252 12233 raft_consensus.cc:2781] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:39:25.314707 12233 raft_consensus.cc:686] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [term 1 LEADER]: Becoming Leader. State: Replica: 4a78e5b9682d4419b4c5b2b716efaddc, State: Running, Role: LEADER
I20260928 08:39:25.315171 12233 consensus_queue.cc:237] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e7bc01c5f54731ab1fff9a963f6e08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45885 } } peers { permanent_uuid: "4a78e5b9682d4419b4c5b2b716efaddc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38807 } } peers { permanent_uuid: "f3fd0679742f47a883959801ac42f637" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38153 } }
I20260928 08:39:25.319507 11715 catalog_manager.cc:5495] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a78e5b9682d4419b4c5b2b716efaddc (127.0.97.3). New cstate: current_term: 1 leader_uuid: "4a78e5b9682d4419b4c5b2b716efaddc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e7bc01c5f54731ab1fff9a963f6e08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a78e5b9682d4419b4c5b2b716efaddc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3fd0679742f47a883959801ac42f637" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38153 } health_report { overall_health: UNKNOWN } } }
I20260928 08:39:25.339617 12222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:25.334747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56468 (local address 127.0.97.3:38807)
0928 08:39:25.334912 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 08:39:25.334917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:39:25.334933 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:39:25.335069 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:39:25.335073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:39:25.335142 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:39:25.335258 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:39:25.335558 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.336236 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.337669 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.338056 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.338149 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:25.338631 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:39:25.339088 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:39:25.339198 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 08:39:25.339289 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 08:39:25.339344 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 08:39:25.347618 11884 raft_consensus.cc:1261] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637 [term 1 FOLLOWER]: Refusing update from remote peer 4a78e5b9682d4419b4c5b2b716efaddc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:39:25.347623 12027 raft_consensus.cc:1261] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08 [term 1 FOLLOWER]: Refusing update from remote peer 4a78e5b9682d4419b4c5b2b716efaddc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:39:25.348495 12235 consensus_queue.cc:1035] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7e7bc01c5f54731ab1fff9a963f6e08" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:39:25.348870 12233 consensus_queue.cc:1035] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3fd0679742f47a883959801ac42f637" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 08:39:25.357753 12073 tablet.cc:2369] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:39:25.359644 12238 mvcc.cc:205] Tried to move back new op lower bound from 7334235198853861376 to 7334235198734602240. Current Snapshot: MvccSnapshot[applied={T|T < 7334235198853861376}]
I20260928 08:39:25.374096 11715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:39:25.374382 11715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:39:25.377811 11715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7156a0280ed45c59ffc4c09e36dc707: Sending DeleteTablet for 3 replicas of tablet 78e8c905597a4874ae12915ed74d1ce7
I20260928 08:39:25.378903 12007 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e8c905597a4874ae12915ed74d1ce7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:39:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44142
I20260928 08:39:25.378916 12141 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e8c905597a4874ae12915ed74d1ce7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:39:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56466
I20260928 08:39:25.379371 11864 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e8c905597a4874ae12915ed74d1ce7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:39:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37540
I20260928 08:39:25.382066 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 11796
I20260928 08:39:25.389828 12249 tablet_replica.cc:330] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc: stopping tablet replica
I20260928 08:39:25.390410 12249 raft_consensus.cc:2229] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:39:25.391199 12249 raft_consensus.cc:2258] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:39:25.393426 12249 ts_tablet_manager.cc:1905] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:39:25.400971 12249 ts_tablet_manager.cc:1918] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:39:25.401237 12249 log.cc:1196] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/78e8c905597a4874ae12915ed74d1ce7
I20260928 08:39:25.401873 12249 ts_tablet_manager.cc:1939] T 78e8c905597a4874ae12915ed74d1ce7 P 4a78e5b9682d4419b4c5b2b716efaddc: Deleting consensus metadata
I20260928 08:39:25.403822 11701 catalog_manager.cc:4841] TS 4a78e5b9682d4419b4c5b2b716efaddc (127.0.97.3:38807): tablet 78e8c905597a4874ae12915ed74d1ce7 (table test-table [id=3d7b3e9dc86d488994e355eebd26e647]) successfully deleted
I20260928 08:39:25.407895 12250 tablet_replica.cc:330] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637: stopping tablet replica
I20260928 08:39:25.408447 12250 raft_consensus.cc:2229] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:39:25.409246 12250 raft_consensus.cc:2258] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:39:25.409844 12248 tablet_replica.cc:330] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08: stopping tablet replica
I20260928 08:39:25.410264 12248 raft_consensus.cc:2229] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:39:25.410807 12248 raft_consensus.cc:2258] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:39:25.412248 12250 ts_tablet_manager.cc:1905] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:39:25.412783 12248 ts_tablet_manager.cc:1905] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 08:39:25 dist-test-slave-w6c0 krb5kdc[11240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:39:25.419795 12248 ts_tablet_manager.cc:1918] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:39:25.420044 12248 log.cc:1196] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/78e8c905597a4874ae12915ed74d1ce7
I20260928 08:39:25.420069 12250 ts_tablet_manager.cc:1918] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:39:25.420292 12250 log.cc:1196] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584618551208-388-0/minicluster-data/ts-0/wal/wals/78e8c905597a4874ae12915ed74d1ce7
I20260928 08:39:25.420629 12248 ts_tablet_manager.cc:1939] T 78e8c905597a4874ae12915ed74d1ce7 P b7e7bc01c5f54731ab1fff9a963f6e08: Deleting consensus metadata
I20260928 08:39:25.420899 12250 ts_tablet_manager.cc:1939] T 78e8c905597a4874ae12915ed74d1ce7 P f3fd0679742f47a883959801ac42f637: Deleting consensus metadata
I20260928 08:39:25.422701 11702 catalog_manager.cc:4841] TS b7e7bc01c5f54731ab1fff9a963f6e08 (127.0.97.2:45885): tablet 78e8c905597a4874ae12915ed74d1ce7 (table test-table [id=3d7b3e9dc86d488994e355eebd26e647]) successfully deleted
I20260928 08:39:25.422995 11702 catalog_manager.cc:4841] TS f3fd0679742f47a883959801ac42f637 (127.0.97.1:38153): tablet 78e8c905597a4874ae12915ed74d1ce7 (table test-table [id=3d7b3e9dc86d488994e355eebd26e647]) successfully deleted
I20260928 08:39:25.427014 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:25.383774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.0.97.1:38153)
0928 08:39:25.384012 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 08:39:25.384020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:39:25.406990 (+ 22970us) server_negotiation.cc:407] Connection header received
0928 08:39:25.407413 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:39:25.407425 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:39:25.407583 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:39:25.407738 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:39:25.410221 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.411404 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.413292 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.413974 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.414078 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:25.420125 (+ 6047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:39:25.420139 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:39:25.420157 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:39:25.420209 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:39:25.424030 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:39:25.424776 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:39:25.424785 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:39:25.424788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:39:25.424915 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:39:25.425309 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:39:25.425316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:39:25.425319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:39:25.425682 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:39:25.426050 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0928 08:39:25.426352 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 08:39:25.426602 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:39:25.427606 11924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:39:25.539784 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 11796
I20260928 08:39:25.557760 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 11935
Sep 28 08:39:25 dist-test-slave-w6c0 krb5kdc[11240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:39:25.574659 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:25.558626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44154 (local address 127.0.97.2:45885)
0928 08:39:25.558794 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 08:39:25.558798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:39:25.558816 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:39:25.559005 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:39:25.559010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:39:25.559073 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:39:25.559164 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:39:25.560787 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.561483 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.563446 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.563845 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.563943 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:25.568914 (+ 4971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:39:25.568930 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:39:25.568946 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:39:25.568980 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:39:25.571848 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:39:25.572541 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:39:25.572548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:39:25.572550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:39:25.572672 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:39:25.573070 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:39:25.573079 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:39:25.573081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:39:25.573446 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:39:25.573763 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 08:39:25.574056 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 08:39:25.574257 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:39:25.575351 12067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:39:25.671849 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 11935
I20260928 08:39:25.689316 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 12078
Sep 28 08:39:25 dist-test-slave-w6c0 krb5kdc[11240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:39:25.706815 12222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:39:25.690220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56478 (local address 127.0.97.3:38807)
0928 08:39:25.690408 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 08:39:25.690413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:39:25.690433 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:39:25.690639 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:39:25.690643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:39:25.690715 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:39:25.690804 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:39:25.692619 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.693416 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.695449 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:39:25.695938 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:39:25.696036 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:39:25.700622 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:39:25.700635 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:39:25.700650 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:39:25.700687 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:39:25.703800 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:39:25.704629 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:39:25.704636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:39:25.704638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:39:25.704786 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:39:25.705237 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:39:25.705241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:39:25.705247 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:39:25.705690 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:39:25.705902 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 08:39:25.706296 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 08:39:25.706450 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:39:25.707547 12201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:39:25.832130 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 12078
I20260928 08:39:25.851539 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 11684
I20260928 08:39:25.852715 11746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:39:25.965277 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 11684
I20260928 08:39:25.986656 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 08:39:26.321947 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 08:39:26.322155 388 mini_ranger.cc:66] Stopping Ranger...
I20260928 08:39:26.667405 388 mini_ranger.cc:68] Stopped Ranger
2026-09-28 08:39:26.667 UTC [11270] LOG: received smart shutdown request
2026-09-28 08:39:26.671 UTC [11270] LOG: background worker "logical replication launcher" (PID 11279) exited with exit code 1
2026-09-28 08:39:26.671 UTC [11274] LOG: shutting down
2026-09-28 08:39:26.840 UTC [11270] LOG: database system is shut down
2026-09-28T08:39:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49667 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:39:27 dist-test-slave-w6c0 krb5kdc[12264](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:39:27 dist-test-slave-w6c0 krb5kdc[12264](info): set up 2 sockets
Sep 28 08:39:27 dist-test-slave-w6c0 krb5kdc[12264](info): commencing operation
krb5kdc: starting...
W20260928 08:39:28.070987 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:39:28 dist-test-slave-w6c0 krb5kdc[12264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:39:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:39:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260928 08:39:28.387660 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task38vn8d/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/postgres -l logfile start
2026-09-28 08:39:30.827 UTC [12294] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 08:39:30.827 UTC [12294] LOG: listening on IPv4 address "127.0.97.20", port 57675
2026-09-28 08:39:30.831 UTC [12294] LOG: listening on Unix socket "/tmp/.s.PGSQL.57675"
2026-09-28 08:39:30.852 UTC [12297] LOG: database system was shut down at 2026-09-28 08:39:29 UTC
2026-09-28 08:39:30.856 UTC [12294] LOG: database system is ready to accept connections
I20260928 08:39:31.826763 388 mini_postgres.cc:96] Postgres bound to 57675
2026-09-28 08:39:31.843 UTC [12305] FATAL: database "slave" does not exist
127.0.97.20:57675 - accepting connections
I20260928 08:39:31.844297 388 mini_ranger.cc:161] Starting Ranger...
I20260928 08:39:31.869032 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 08:39:32.351179 388 mini_ranger.cc:87] Created ranger Postgres database
I20260928 08:39:32.351311 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ranger-admin
2026-09-28 08:39:32,650 [I] DB FLAVOR :POSTGRES
2026-09-28 08:39:32,650 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 08:39:32,650 [I] Checking connection..
2026-09-28 08:39:32,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 08:39:32,884 [I] Checking connection passed.
2026-09-28 08:39:32,884 [I] --------- Verifying version history table ---------
2026-09-28 08:39:32,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:39:33,148 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 08:39:33,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:39:33,413 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 08:39:33,414 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 08:39:33,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 08:39:33.656 UTC [12369] WARNING: there is no transaction in progress
2026-09-28 08:39:33,663 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 08:39:33,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:39:33,909 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 08:39:33,910 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 08:39:33,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 08:39:34,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 08:39:34,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-w6c0.c.gcp-upstream.internal','N') ;"
2026-09-28 08:39:34,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 08:39:34,876 [I] Table x_portal_user does not exist in database ranger
2026-09-28 08:39:34,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 08:39:35,129 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 08:39:35,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 08:39:35,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 08:39:35,609 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 08:39:35,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 08:39:36.561 UTC [12477] WARNING: there is no transaction in progress
2026-09-28 08:39:36.577 UTC [12477] WARNING: there is no transaction in progress
2026-09-28 08:39:36.594 UTC [12477] WARNING: there is no transaction in progress
2026-09-28 08:39:36.609 UTC [12477] WARNING: there is no transaction in progress
2026-09-28 08:39:36.625 UTC [12477] WARNING: there is no transaction in progress
2026-09-28 08:39:36.646 UTC [12477] WARNING: there is no transaction in progress
2026-09-28 08:39:36.698 UTC [12477] WARNING: there is no transaction in progress
2026-09-28 08:39:36.704 UTC [12477] WARNING: there is no transaction in progress
2026-09-28 08:39:36.718 UTC [12477] WARNING: there is no transaction in progress
2026-09-28 08:39:36.728 UTC [12477] WARNING: there is no transaction in progress
2026-09-28 08:39:36.738 UTC [12477] WARNING: there is no transaction in progress
2026-09-28 08:39:36.752 UTC [12477] WARNING: there is no transaction in progress
2026-09-28 08:39:36.761 UTC [12477] WARNING: there is no transaction in progress
2026-09-28 08:39:36.767 UTC [12477] WARNING: there is no transaction in progress
2026-09-28 08:39:36.773 UTC [12477] WARNING: there is no transaction in progress
2026-09-28 08:39:36.780 UTC [12477] WARNING: there is no transaction in progress
2026-09-28 08:39:37,402 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 08:39:37,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 08:39:37,622 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 08:39:37,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 08:39:37,873 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 08:39:37,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 08:39:38,136 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 08:39:38,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 08:39:38,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 08:39:38,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-w6c0.c.gcp-upstream.internal';"
2026-09-28 08:39:38,892 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 08:39:38,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 08:39:39,115 [I] DB_PATCHES have already been applied
I20260928 08:39:39.120972 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task38vn8d/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 08:39:39.121059 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260928 08:39:39.135408 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:46031
Sep 28, 2026 8:39:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:39:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:39:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 8:39:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 8:39:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 8:39:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 8:39:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 08:39:40.268137 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.424s user 0.001s sys 0.035s
Sep 28, 2026 8:40:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 8:40:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 8:40:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 8:40:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 8:40:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 8:40:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 8:40:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 8:40:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 8:40:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 8:40:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 8:40:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 8:40:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 8:40:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 8:40:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 8:40:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 8:40:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 8:40:01 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 8:40:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 08:40:03.096408 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 08:40:03 dist-test-slave-w6c0 krb5kdc[12264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584803, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 08:40:03.258343 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 08:40:03.278388 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 08:40:04.154579 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 08:40:04.154695 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ranger-kms
I20260928 08:40:04.154745 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:57675
2026-09-28 08:40:04,499 [I] DB FLAVOR :POSTGRES
2026-09-28 08:40:04,499 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 08:40:04,499 [I] Checking connection
2026-09-28 08:40:04,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 08:40:04,744 [I] connection success
2026-09-28 08:40:04,744 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 08:40:04,744 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 08:40:04,744 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 08:40:04,974 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 08:40:04,974 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 08:40:04,974 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 08:40:04,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57675/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 08:40:05,219 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 08:40:05.680701 388 mini_ranger_kms.cc:310] Created kms service
I20260928 08:40:05.867877 388 mini_ranger_kms.cc:326] Created kudu user
I20260928 08:40:05.911614 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 08:40:05 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584803, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:40:06.340183 388 mini_ranger_kms.cc:384] Added ranger policy
I20260928 08:40:06.340503 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task38vn8d/build/asan/bin/postgresql.jar:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task38vn8d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 08:40:06.340550 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260928 08:40:06.358188 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12658
I20260928 08:40:06.358413 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:37325
Sep 28, 2026 8:40:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:40:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:40:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 8:40:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 8:40:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 8:40:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 08:40:08.656797 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.398s user 0.068s sys 0.295s
I20260928 08:40:08.657044 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 08:40:08.657204 388 mini_ranger_kms.cc:401] 127.0.97.20:37325/kms/v1/keys
Sep 28 08:40:08 dist-test-slave-w6c0 krb5kdc[12264](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790584808, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 08:40:09 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584808, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 28 08:40:09 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584808, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 28 08:40:11 dist-test-slave-w6c0 krb5kdc[12264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:40:11.848385 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39755
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39755
--ranger_config_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:37325/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:40:12.182916 12708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:12.183286 12708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:12.183441 12708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:12.191000 12708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:40:12.191171 12708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:12.191270 12708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:12.191370 12708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:40:12.191459 12708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:40:12.202250 12708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:37325/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39755
--ranger_config_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:12.203668 12708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:12.205051 12708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:12.219266 12714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:12.219873 12716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:12.219256 12713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:12.220299 12708 server_base.cc:1034] running on GCE node
I20260928 08:40:12.221338 12708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:12.223213 12708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:12.224466 12708 hybrid_clock.cc:648] HybridClock initialized: now 1790584812224399 us; error 57 us; skew 500 ppm
I20260928 08:40:12.224884 12708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:12 dist-test-slave-w6c0 krb5kdc[12264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:12.237293 12708 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:40:12.239300 12708 webserver.cc:460] Webserver started at http://127.0.97.62:45121/ using document root <none> and password file <none>
I20260928 08:40:12.240227 12708 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:12.240428 12708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:12.240787 12708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:40:12 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:40:12.302577 12708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6"
format_stamp: "Formatted at 2026-09-28 08:40:12 on dist-test-slave-w6c0"
server_key: "f9d600b1f59c5f2bf5a9838262134589"
server_key_iv: "af354f84d5aa7194f497fc21fb192ae9"
server_key_version: "kuduclusterkey@0"
I20260928 08:40:12.303525 12708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6"
format_stamp: "Formatted at 2026-09-28 08:40:12 on dist-test-slave-w6c0"
server_key: "f9d600b1f59c5f2bf5a9838262134589"
server_key_iv: "af354f84d5aa7194f497fc21fb192ae9"
server_key_version: "kuduclusterkey@0"
I20260928 08:40:12.309208 12708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260928 08:40:12.329936 12723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:12.331620 12708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:40:12.331812 12708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6"
format_stamp: "Formatted at 2026-09-28 08:40:12 on dist-test-slave-w6c0"
server_key: "f9d600b1f59c5f2bf5a9838262134589"
server_key_iv: "af354f84d5aa7194f497fc21fb192ae9"
server_key_version: "kuduclusterkey@0"
I20260928 08:40:12.332036 12708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:12.366999 12708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:12.371536 12708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:12.371889 12708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:12.392576 12708 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39755
I20260928 08:40:12.392594 12774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39755 every 8 connection(s)
I20260928 08:40:12.394181 12708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:40:12.397358 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 12708
I20260928 08:40:12.397586 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:40:12.398414 388 external_mini_cluster.cc:1428] Setting key d3fc2a9bdfb67501df83a9a848396fa3
I20260928 08:40:12.399040 12775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:40:12.418684 12775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6: Bootstrap starting.
I20260928 08:40:12.422686 12775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6: Neither blocks nor log segments found. Creating new log.
I20260928 08:40:12.424196 12775 log.cc:824] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:12.429540 12775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6: No bootstrap required, opened a new log
Sep 28 08:40:12 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:12.440953 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:12.400649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59946 (local address 127.0.97.62:39755)
0928 08:40:12.415386 (+ 14737us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:12.415405 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:12.415471 (+ 66us) server_negotiation.cc:407] Connection header received
0928 08:40:12.418028 (+ 2557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:12.418063 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:12.418693 (+ 630us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:12.419182 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:12.420869 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:12.422956 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:12.424908 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:12.425893 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:12.426087 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:12.431117 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:12.431130 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:12.431153 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:12.431201 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:12.435538 (+ 4337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:12.436576 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:12.436585 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:12.436595 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:12.436833 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:12.437283 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:12.437288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:12.437291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:12.438130 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:12.438460 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:12.438963 (+ 503us) server_negotiation.cc:299] Negotiation successful
0928 08:40:12.439460 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14466,"thread_start_us":17805,"threads_started":1}
I20260928 08:40:12.440953 12775 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39755 } }
I20260928 08:40:12.441505 12775 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:40:12.441615 12775 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0bfbc72e1d14f65b9a5c62ab8a52ff6, State: Initialized, Role: FOLLOWER
I20260928 08:40:12.442323 12775 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39755 } }
I20260928 08:40:12.442685 12775 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:40:12.442847 12775 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:40:12.443043 12775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:40:12.445849 12775 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39755 } }
I20260928 08:40:12.446460 12775 leader_election.cc:302] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0bfbc72e1d14f65b9a5c62ab8a52ff6; no voters:
I20260928 08:40:12.448086 12775 leader_election.cc:288] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:40:12.448189 12780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:40:12.451089 12775 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:40:12.451043 12780 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [term 1 LEADER]: Becoming Leader. State: Replica: e0bfbc72e1d14f65b9a5c62ab8a52ff6, State: Running, Role: LEADER
I20260928 08:40:12.451694 12780 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39755 } }
I20260928 08:40:12.456475 12782 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0bfbc72e1d14f65b9a5c62ab8a52ff6. Latest consensus state: current_term: 1 leader_uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39755 } } }
I20260928 08:40:12.456971 12782 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [sys.catalog]: This master's current role is: LEADER
I20260928 08:40:12.457690 12781 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39755 } } }
I20260928 08:40:12.458034 12781 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [sys.catalog]: This master's current role is: LEADER
I20260928 08:40:12.468351 12786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:40:12.474416 12786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:40:12.493965 12786 catalog_manager.cc:1347] Generated new cluster ID: 3e207a990b7742b68ee37ceaa7ddf084
I20260928 08:40:12.494110 12786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:40:12.507628 12786 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:40:12.507834 12775 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 08:40:12.509521 12786 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:40:12.521502 12786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6: Generated new TSK 0
I20260928 08:40:12.522624 12786 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:40:13.363615 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:37325/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39755
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:40:13 dist-test-slave-w6c0 krb5kdc[12264](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790584813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 08:40:13.761936 12821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:13.762260 12821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:13.762439 12821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:13.771380 12821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:13.771538 12821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:13.771711 12821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:40:13.783020 12821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:37325/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:13.784632 12821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:13.786115 12821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:13.805167 12831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:13.806325 12832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:13.814662 12834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:13.815030 12821 server_base.cc:1034] running on GCE node
I20260928 08:40:13.815932 12821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:13.817049 12821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:13.818393 12821 hybrid_clock.cc:648] HybridClock initialized: now 1790584813818331 us; error 50 us; skew 500 ppm
I20260928 08:40:13.818849 12821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:13 dist-test-slave-w6c0 krb5kdc[12264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:13.823397 12821 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:40:13.847035 12821 webserver.cc:460] Webserver started at http://127.0.97.1:36071/ using document root <none> and password file <none>
I20260928 08:40:13.848268 12821 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:13.848508 12821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:13.848970 12821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:40:13 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:40:13.875308 12821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "b2a9685e507044d88776f70d802886cf"
format_stamp: "Formatted at 2026-09-28 08:40:13 on dist-test-slave-w6c0"
server_key: "551c5b478476b4909ee2d32576d9e1d1"
server_key_iv: "b76614ccae8204c1adafca7d632f92b6"
server_key_version: "kuduclusterkey@0"
I20260928 08:40:13.877220 12821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b2a9685e507044d88776f70d802886cf"
format_stamp: "Formatted at 2026-09-28 08:40:13 on dist-test-slave-w6c0"
server_key: "551c5b478476b4909ee2d32576d9e1d1"
server_key_iv: "b76614ccae8204c1adafca7d632f92b6"
server_key_version: "kuduclusterkey@0"
I20260928 08:40:13.918661 12821 fs_manager.cc:691] Time spent creating directory manager: real 0.041s user 0.019s sys 0.015s
I20260928 08:40:13.944628 12841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:13.946666 12821 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.009s sys 0.000s
I20260928 08:40:13.946892 12821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "b2a9685e507044d88776f70d802886cf"
format_stamp: "Formatted at 2026-09-28 08:40:13 on dist-test-slave-w6c0"
server_key: "551c5b478476b4909ee2d32576d9e1d1"
server_key_iv: "b76614ccae8204c1adafca7d632f92b6"
server_key_version: "kuduclusterkey@0"
I20260928 08:40:13.947152 12821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:13.984140 12821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:13.990026 12821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:13.990387 12821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:14.002394 12821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:40:14.007475 12821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:40:14.007607 12821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:14.007747 12821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:40:14.007833 12821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:14.070863 12821 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43431
I20260928 08:40:14.072268 12953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43431 every 8 connection(s)
I20260928 08:40:14.072613 12821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:40:14.076001 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 12821
I20260928 08:40:14.076256 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:40:14.077339 388 external_mini_cluster.cc:1428] Setting key 7f36716dae5c9ebab4c8f90f5cf3cbfb
Sep 28 08:40:14 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 08:40:14.140496 12956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:14.076122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39755 (local address 127.0.97.1:40143)
0928 08:40:14.100055 (+ 23933us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:14.100124 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:14.103858 (+ 3734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:14.104981 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:14.104994 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:14.105674 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:14.106551 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:14.106576 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:14.113429 (+ 6853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:14.113436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:14.115558 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:14.115583 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:14.126774 (+ 11191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:14.126784 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:14.127156 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:14.128191 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:14.128222 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:14.131033 (+ 2811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:14.135437 (+ 4404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:14.135449 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:14.135470 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:14.136131 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:14.136690 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:14.136696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:14.136700 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:14.136958 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:14.137658 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:14.137666 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:14.138119 (+ 453us) client_negotiation.cc:769] Sending connection context
0928 08:40:14.138420 (+ 301us) client_negotiation.cc:240] Negotiation successful
0928 08:40:14.139018 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23560,"thread_start_us":26376,"threads_started":1}
I20260928 08:40:14.142168 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:14.076925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40143 (local address 127.0.97.62:39755)
0928 08:40:14.099335 (+ 22410us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:14.099342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:14.102571 (+ 3229us) server_negotiation.cc:407] Connection header received
0928 08:40:14.104215 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:14.104221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:14.104326 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:14.104419 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:14.111442 (+ 7023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:14.112611 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:14.115922 (+ 3311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:14.116598 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:14.116729 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:14.131389 (+ 14660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:14.131397 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:14.131414 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:14.131460 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:14.135214 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:14.136352 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:14.136358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:14.136361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:14.136548 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:14.137116 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:14.137128 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:14.137132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:14.137500 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:14.141354 (+ 3854us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:14.141515 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 08:40:14.141766 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22246,"thread_start_us":22434,"threads_started":1}
I20260928 08:40:14.143947 12954 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39755
I20260928 08:40:14.144302 12954 heartbeater.cc:464] Registering TS with master...
I20260928 08:40:14.145198 12954 heartbeater.cc:511] Master 127.0.97.62:39755 requested a full tablet report, sending...
I20260928 08:40:14.147877 12740 ts_manager.cc:194] Registered new tserver with Master: b2a9685e507044d88776f70d802886cf (127.0.97.1:43431)
I20260928 08:40:14.150015 12740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:40:14.176496 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:37325/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39755
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:40:14 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 28 08:40:14 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260928 08:40:14.675981 12960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:14.676436 12960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:14.676606 12960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:14.688635 12960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:14.688825 12960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:14.689019 12960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:40:14.705618 12960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:37325/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:14.707551 12960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:14.709446 12960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:14.737620 12967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:14.741323 12968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:14.747141 12960 server_base.cc:1034] running on GCE node
W20260928 08:40:14.747732 12970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:14.748638 12960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:14.756170 12960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:14.757638 12960 hybrid_clock.cc:648] HybridClock initialized: now 1790584814757526 us; error 118 us; skew 500 ppm
I20260928 08:40:14.758074 12960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:14 dist-test-slave-w6c0 krb5kdc[12264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:14.762638 12960 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:40:14.776773 12960 webserver.cc:460] Webserver started at http://127.0.97.2:35309/ using document root <none> and password file <none>
I20260928 08:40:14.777930 12960 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:14.778103 12960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:14.778574 12960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:40:14 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:40:14.810369 12960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "70cecfed712b46b288d01dd990d4f2b4"
format_stamp: "Formatted at 2026-09-28 08:40:14 on dist-test-slave-w6c0"
server_key: "39f24a9b3d7f2becd34eae7b5cd9e9d0"
server_key_iv: "7443341930e7592f293680c57802e045"
server_key_version: "kuduclusterkey@0"
I20260928 08:40:14.812590 12960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "70cecfed712b46b288d01dd990d4f2b4"
format_stamp: "Formatted at 2026-09-28 08:40:14 on dist-test-slave-w6c0"
server_key: "39f24a9b3d7f2becd34eae7b5cd9e9d0"
server_key_iv: "7443341930e7592f293680c57802e045"
server_key_version: "kuduclusterkey@0"
I20260928 08:40:14.838902 12960 fs_manager.cc:691] Time spent creating directory manager: real 0.026s user 0.014s sys 0.012s
I20260928 08:40:14.870675 12978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:14.872879 12960 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.006s sys 0.008s
I20260928 08:40:14.873113 12960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "70cecfed712b46b288d01dd990d4f2b4"
format_stamp: "Formatted at 2026-09-28 08:40:14 on dist-test-slave-w6c0"
server_key: "39f24a9b3d7f2becd34eae7b5cd9e9d0"
server_key_iv: "7443341930e7592f293680c57802e045"
server_key_version: "kuduclusterkey@0"
I20260928 08:40:14.873397 12960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:14.905236 12960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:14.910887 12960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:14.911268 12960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:14.913475 12960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:40:14.915522 12960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:40:14.915624 12960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:14.915758 12960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:40:14.915820 12960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:14.958523 12960 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41059
I20260928 08:40:14.958544 13096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41059 every 8 connection(s)
I20260928 08:40:14.959930 12960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:40:14.970201 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 12960
I20260928 08:40:14.970409 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:40:14.971128 388 external_mini_cluster.cc:1428] Setting key 13d860b1175501c6f964845176f3c3fa
Sep 28 08:40:14 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:15.004247 13099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:14.963244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52933 (local address 127.0.97.62:39755)
0928 08:40:14.978575 (+ 15331us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:14.978581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:14.978612 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:40:14.982707 (+ 4095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:14.982715 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:14.982836 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:14.982943 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:14.985296 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:14.986561 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:14.989281 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:14.990036 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:14.990180 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:14.995350 (+ 5170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:14.995360 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:14.995368 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:14.995424 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:14.999659 (+ 4235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:15.000851 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:15.000860 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:15.000866 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:15.001032 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:15.001590 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:15.001598 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:15.001605 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:15.002051 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:15.002217 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:15.003386 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0928 08:40:15.003709 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15172,"thread_start_us":15371,"threads_started":1}
I20260928 08:40:15.005404 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:14.964032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39755 (local address 127.0.97.2:52933)
0928 08:40:14.977838 (+ 13806us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:14.977917 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:14.982143 (+ 4226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:14.983392 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:14.983407 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:14.984121 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:14.985058 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:14.985088 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:14.986757 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:14.986765 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:14.989044 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:14.989073 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:14.990538 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:14.990549 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:14.990903 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:14.991759 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:14.991794 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:14.994996 (+ 3202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:14.999887 (+ 4891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:14.999902 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:14.999927 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:15.000686 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:15.001171 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:15.001178 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:15.001185 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:15.001447 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:15.002280 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:15.002300 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:15.002888 (+ 588us) client_negotiation.cc:769] Sending connection context
0928 08:40:15.003239 (+ 351us) client_negotiation.cc:240] Negotiation successful
0928 08:40:15.003876 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13319,"thread_start_us":16256,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 08:40:15.007534 13097 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39755
I20260928 08:40:15.007961 13097 heartbeater.cc:464] Registering TS with master...
I20260928 08:40:15.009009 13097 heartbeater.cc:511] Master 127.0.97.62:39755 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 08:40:15.011058 12740 ts_manager.cc:194] Registered new tserver with Master: 70cecfed712b46b288d01dd990d4f2b4 (127.0.97.2:41059)
I20260928 08:40:15.012435 12740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:40:15.047096 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:37325/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39755
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:40:15.154343 12954 heartbeater.cc:503] Master 127.0.97.62:39755 was elected leader, sending a full tablet report...
W20260928 08:40:15.415345 13103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:15.415673 13103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:15.415836 13103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:15.423512 13103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:15.423625 13103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:15.423818 13103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:40:15.433965 13103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:37325/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:15.435447 13103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:15.436803 13103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:15.450587 13109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:15.450651 13108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:15.450621 13111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:15.456573 13103 server_base.cc:1034] running on GCE node
I20260928 08:40:15.457226 13103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:15.458220 13103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:15.459468 13103 hybrid_clock.cc:648] HybridClock initialized: now 1790584815459389 us; error 69 us; skew 500 ppm
I20260928 08:40:15.459764 13103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:15 dist-test-slave-w6c0 krb5kdc[12264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:15.463230 13103 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:40:15.464808 13103 webserver.cc:460] Webserver started at http://127.0.97.3:37921/ using document root <none> and password file <none>
I20260928 08:40:15.465552 13103 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:15.465653 13103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:15.465907 13103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:40:15 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:40:15.481819 13103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "a33a09985dbe4cb6b8388c5ff38aad0e"
format_stamp: "Formatted at 2026-09-28 08:40:15 on dist-test-slave-w6c0"
server_key: "d9516da259dcf201bb5dabae5e5f0c5e"
server_key_iv: "2db037f1111d7c248a2998482b4e19dc"
server_key_version: "kuduclusterkey@0"
I20260928 08:40:15.482908 13103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a33a09985dbe4cb6b8388c5ff38aad0e"
format_stamp: "Formatted at 2026-09-28 08:40:15 on dist-test-slave-w6c0"
server_key: "d9516da259dcf201bb5dabae5e5f0c5e"
server_key_iv: "2db037f1111d7c248a2998482b4e19dc"
server_key_version: "kuduclusterkey@0"
I20260928 08:40:15.489332 13103 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 08:40:15.509533 13118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:15.511235 13103 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 08:40:15.511451 13103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "a33a09985dbe4cb6b8388c5ff38aad0e"
format_stamp: "Formatted at 2026-09-28 08:40:15 on dist-test-slave-w6c0"
server_key: "d9516da259dcf201bb5dabae5e5f0c5e"
server_key_iv: "2db037f1111d7c248a2998482b4e19dc"
server_key_version: "kuduclusterkey@0"
I20260928 08:40:15.511667 13103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:15.544831 13103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:15.548911 13103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:15.549237 13103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:15.550207 13103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:40:15.552011 13103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:40:15.552089 13103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:15.552182 13103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:40:15.552249 13103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:15.588040 13103 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35191
I20260928 08:40:15.588094 13230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35191 every 8 connection(s)
I20260928 08:40:15.589370 13103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:40:15.594600 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 13103
I20260928 08:40:15.594767 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:40:15.595291 388 external_mini_cluster.cc:1428] Setting key f37b478873f6d82b9177818474752674
Sep 28 08:40:15 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:15.612159 13233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:15.591989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39279 (local address 127.0.97.62:39755)
0928 08:40:15.592532 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:15.592539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:15.593397 (+ 858us) server_negotiation.cc:407] Connection header received
0928 08:40:15.595476 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:15.595482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:15.595575 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:15.595699 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:15.598066 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:15.598959 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:15.601302 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:15.601759 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:15.601864 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:15.605762 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:15.605768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:15.605771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:15.605822 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:15.608659 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:15.609694 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:15.609698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:15.609702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:15.609874 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:15.610423 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:15.610430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:15.610436 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:15.610861 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:15.611029 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:15.611714 (+ 685us) server_negotiation.cc:299] Negotiation successful
0928 08:40:15.611904 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":252,"threads_started":1}
I20260928 08:40:15.613200 13234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:15.592491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39755 (local address 127.0.97.3:39279)
0928 08:40:15.593239 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:15.593281 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:15.595157 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:15.595964 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:15.595978 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:15.596568 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:15.597235 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:15.597260 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:15.599357 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:15.599362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:15.601141 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:15.601154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:15.601971 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:15.601976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:15.602177 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:15.603169 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:15.603193 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:15.605493 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:15.608876 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:15.608888 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:15.608908 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:15.609519 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:15.609999 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:15.610005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:15.610012 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:15.610291 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:15.611009 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:15.611017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:15.611381 (+ 364us) client_negotiation.cc:769] Sending connection context
0928 08:40:15.611624 (+ 243us) client_negotiation.cc:240] Negotiation successful
0928 08:40:15.612033 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":302,"threads_started":1}
I20260928 08:40:15.615099 13231 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39755
I20260928 08:40:15.615396 13231 heartbeater.cc:464] Registering TS with master...
I20260928 08:40:15.616127 13231 heartbeater.cc:511] Master 127.0.97.62:39755 requested a full tablet report, sending...
I20260928 08:40:15.617655 12740 ts_manager.cc:194] Registered new tserver with Master: a33a09985dbe4cb6b8388c5ff38aad0e (127.0.97.3:35191)
I20260928 08:40:15.618600 12740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39279
I20260928 08:40:15.623869 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:40:15.625563 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 12821
Sep 28 08:40:15 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:40:15.645852 13236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:15.626640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60660 (local address 127.0.97.1:43431)
0928 08:40:15.627325 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:15.627334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:15.627505 (+ 171us) server_negotiation.cc:407] Connection header received
0928 08:40:15.627916 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:15.627928 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:15.628294 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:15.628515 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:15.630713 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:15.631870 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:15.633447 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:15.634037 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:15.634141 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:15.639215 (+ 5074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:15.639230 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:15.639247 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:15.639295 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:15.642245 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:15.643244 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:15.643258 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:15.643260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:15.643541 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:15.644183 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:15.644188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:15.644191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:15.644645 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:15.644915 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:15.645330 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 08:40:15.645562 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":339,"threads_started":1}
I20260928 08:40:15.647313 12949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:15.731670 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 12821
I20260928 08:40:15.748952 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 12960
Sep 28 08:40:15 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:40:15.766458 13238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:15.749869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.0.97.2:41059)
0928 08:40:15.750427 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:15.750434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:15.750458 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:40:15.750580 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:15.750606 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:15.750873 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:15.751046 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:15.752431 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:15.753445 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:15.754994 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:15.755558 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:15.755715 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:15.760422 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:15.760435 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:15.760450 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:15.760496 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:15.763506 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:15.764262 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:15.764275 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:15.764277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:15.764424 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:15.764815 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:15.764824 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:15.764826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:15.765173 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:15.765483 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:15.765797 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 08:40:15.766093 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":282,"threads_started":1}
I20260928 08:40:15.767792 13092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:15.852331 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 12960
I20260928 08:40:15.868515 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 13103
Sep 28 08:40:15 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:40:15.885082 13240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:15.869514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42870 (local address 127.0.97.3:35191)
0928 08:40:15.870077 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:15.870084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:15.870113 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:40:15.870183 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:15.870189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:15.870407 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:15.870609 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:15.871922 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:15.872933 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:15.874249 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:15.875084 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:15.875190 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:15.879598 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:15.879610 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:15.879624 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:15.879662 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:15.882454 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:15.883246 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:15.883254 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:15.883256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:15.883387 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:15.883711 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:15.883714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:15.883716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:15.884039 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:15.884265 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:15.884481 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 08:40:15.884683 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":266,"threads_started":1}
I20260928 08:40:15.886395 13226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:15.969946 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 13103
I20260928 08:40:15.986099 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 12708
I20260928 08:40:15.987504 12770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:16.095239 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 12708
I20260928 08:40:16.118901 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39755
--webserver_interface=127.0.97.62
--webserver_port=45121
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39755
--ranger_config_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:37325/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:37325/kms with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:40:16.457610 13243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:16.457998 13243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:16.458163 13243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:16.465667 13243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:40:16.465860 13243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:16.465983 13243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:16.466082 13243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:40:16.466178 13243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:40:16.476718 13243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:37325/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39755
--ranger_config_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:16.478127 13243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:16.480283 13243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:16.492750 13249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:16.492833 13251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:16.492833 13248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:16.494773 13243 server_base.cc:1034] running on GCE node
I20260928 08:40:16.495615 13243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:16.497126 13243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:16.498370 13243 hybrid_clock.cc:648] HybridClock initialized: now 1790584816498332 us; error 44 us; skew 500 ppm
I20260928 08:40:16.498773 13243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:16 dist-test-slave-w6c0 krb5kdc[12264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:16.502053 13243 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:40:16.503765 13243 webserver.cc:460] Webserver started at http://127.0.97.62:45121/ using document root <none> and password file <none>
I20260928 08:40:16.504467 13243 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:16.504618 13243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 08:40:16 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:40:16.573573 13243 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 08:40:16.576928 13259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:16.578729 13243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:40:16.578923 13243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6"
format_stamp: "Formatted at 2026-09-28 08:40:12 on dist-test-slave-w6c0"
server_key: "f9d600b1f59c5f2bf5a9838262134589"
server_key_iv: "af354f84d5aa7194f497fc21fb192ae9"
server_key_version: "kuduclusterkey@0"
I20260928 08:40:16.580075 13243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:16.602615 13243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:16.606752 13243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:16.607038 13243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:16.625602 13243 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39755
I20260928 08:40:16.625623 13310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39755 every 8 connection(s)
I20260928 08:40:16.627194 13243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:40:16.633950 13311 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:40:16.635452 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 13243
I20260928 08:40:16.636554 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:37325/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43431
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39755
--webserver_port=36071
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:37325/kms with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:40:16.638886 13311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6: Bootstrap starting.
I20260928 08:40:16.659518 13311 log.cc:824] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:16.668490 13311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:40:16.669046 13311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6: Bootstrap complete.
I20260928 08:40:16.678079 13311 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39755 } }
I20260928 08:40:16.679035 13311 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0bfbc72e1d14f65b9a5c62ab8a52ff6, State: Initialized, Role: FOLLOWER
I20260928 08:40:16.679688 13311 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39755 } }
I20260928 08:40:16.679924 13311 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:40:16.680047 13311 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:40:16.680241 13311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:40:16.684310 13311 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39755 } }
I20260928 08:40:16.684835 13311 leader_election.cc:302] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0bfbc72e1d14f65b9a5c62ab8a52ff6; no voters:
I20260928 08:40:16.685508 13311 leader_election.cc:288] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 08:40:16.685652 13315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:40:16.688076 13311 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:40:16.692829 13315 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [term 2 LEADER]: Becoming Leader. State: Replica: e0bfbc72e1d14f65b9a5c62ab8a52ff6, State: Running, Role: LEADER
I20260928 08:40:16.693581 13315 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39755 } }
I20260928 08:40:16.696620 13316 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39755 } } }
I20260928 08:40:16.697082 13316 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [sys.catalog]: This master's current role is: LEADER
I20260928 08:40:16.697705 13317 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0bfbc72e1d14f65b9a5c62ab8a52ff6. Latest consensus state: current_term: 2 leader_uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0bfbc72e1d14f65b9a5c62ab8a52ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39755 } } }
I20260928 08:40:16.698027 13317 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6 [sys.catalog]: This master's current role is: LEADER
I20260928 08:40:16.700174 13326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:40:16.706168 13326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:40:16.706992 13311 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 08:40:16.707890 13326 catalog_manager.cc:1259] Loaded cluster ID: 3e207a990b7742b68ee37ceaa7ddf084
I20260928 08:40:16.708051 13326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:40:16.713246 13326 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:40:16.715250 13326 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6: Loaded TSK: 0
I20260928 08:40:16.717209 13326 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 08:40:17.084582 13313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:17.085089 13313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:17.085286 13313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:17.097306 13313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:17.097484 13313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:17.097728 13313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:40:17.114077 13313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:37325/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36071
--tserver_master_addrs=127.0.97.62:39755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:17.116108 13313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:17.117921 13313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:17.168109 13350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:17.170943 13349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:17.175871 13352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:17.176898 13313 server_base.cc:1034] running on GCE node
I20260928 08:40:17.177945 13313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:17.179493 13313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:17.182935 13313 hybrid_clock.cc:648] HybridClock initialized: now 1790584817182845 us; error 75 us; skew 500 ppm
I20260928 08:40:17.183377 13313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:17 dist-test-slave-w6c0 krb5kdc[12264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:17.187757 13313 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:40:17.227131 13313 webserver.cc:460] Webserver started at http://127.0.97.1:36071/ using document root <none> and password file <none>
I20260928 08:40:17.228412 13313 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:17.228699 13313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 08:40:17 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:40:17.287302 13313 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.000s
I20260928 08:40:17.297346 13361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:17.299340 13313 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.009s sys 0.000s
I20260928 08:40:17.299611 13313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "b2a9685e507044d88776f70d802886cf"
format_stamp: "Formatted at 2026-09-28 08:40:13 on dist-test-slave-w6c0"
server_key: "551c5b478476b4909ee2d32576d9e1d1"
server_key_iv: "b76614ccae8204c1adafca7d632f92b6"
server_key_version: "kuduclusterkey@0"
I20260928 08:40:17.301203 13313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:17.343763 13313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:17.349639 13313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:17.350029 13313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:17.354212 13313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:40:17.359382 13313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:40:17.359534 13313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:17.359649 13313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:40:17.359725 13313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:17.431275 13313 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43431
I20260928 08:40:17.431365 13473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43431 every 8 connection(s)
I20260928 08:40:17.433130 13313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:40:17.443538 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 13313
I20260928 08:40:17.444337 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:37325/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41059
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39755
--webserver_port=35309
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:37325/kms with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:40:17 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:17.463604 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:17.436603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39755 (local address 127.0.97.1:33993)
0928 08:40:17.437480 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:17.437520 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:17.439295 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:17.441732 (+ 2437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:17.441744 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:17.442377 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:17.443223 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:17.443249 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:17.445780 (+ 2531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:17.445786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:17.447887 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:17.447913 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:17.449815 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:17.449823 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:17.450102 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:17.450977 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:17.451011 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:17.454097 (+ 3086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:17.457880 (+ 3783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:17.457890 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:17.457906 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:17.458356 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:17.458920 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:17.458935 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:17.458937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:17.459300 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:17.460632 (+ 1332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:17.460644 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:17.461102 (+ 458us) client_negotiation.cc:769] Sending connection context
0928 08:40:17.461407 (+ 305us) client_negotiation.cc:240] Negotiation successful
0928 08:40:17.462050 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":3564,"threads_started":1}
I20260928 08:40:17.464108 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:17.436403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33993 (local address 127.0.97.62:39755)
0928 08:40:17.437157 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:17.437179 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:17.437660 (+ 481us) server_negotiation.cc:407] Connection header received
0928 08:40:17.440038 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:17.440059 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:17.440577 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:17.440985 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:17.443509 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:17.445540 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:17.448108 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:17.449170 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:17.449371 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:17.454477 (+ 5106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:17.454486 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:17.454546 (+ 60us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:17.454601 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:17.457709 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:17.458563 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:17.458570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:17.458576 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:17.458766 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:17.459480 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:17.459490 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:17.459495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:17.460325 (+ 830us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:17.460551 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:17.462385 (+ 1834us) server_negotiation.cc:299] Negotiation successful
0928 08:40:17.462852 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":308,"threads_started":1}
I20260928 08:40:17.467595 13474 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39755
I20260928 08:40:17.467942 13474 heartbeater.cc:464] Registering TS with master...
I20260928 08:40:17.468931 13474 heartbeater.cc:511] Master 127.0.97.62:39755 requested a full tablet report, sending...
W20260928 08:40:17.472661 13474 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39755 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 08:40:17.472919 13474 heartbeater.cc:464] Registering TS with master...
I20260928 08:40:17.473261 13474 heartbeater.cc:511] Master 127.0.97.62:39755 requested a full tablet report, sending...
I20260928 08:40:17.474823 13474 heartbeater.cc:464] Registering TS with master...
I20260928 08:40:17.475236 13474 heartbeater.cc:511] Master 127.0.97.62:39755 requested a full tablet report, sending...
W20260928 08:40:17.476646 13474 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:40:17.898844 13478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:17.899204 13478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:17.899371 13478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:17.907534 13478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:17.907689 13478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:17.907883 13478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:40:17.918375 13478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:37325/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35309
--tserver_master_addrs=127.0.97.62:39755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:17.919952 13478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:17.921422 13478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:17.940951 13494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:17.942262 13495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:17.950779 13497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:17.951082 13478 server_base.cc:1034] running on GCE node
I20260928 08:40:17.951854 13478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:17.953123 13478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:17.954455 13478 hybrid_clock.cc:648] HybridClock initialized: now 1790584817954380 us; error 64 us; skew 500 ppm
I20260928 08:40:17.954893 13478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:17 dist-test-slave-w6c0 krb5kdc[12264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:17.960271 13478 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
Sep 28 08:40:17 dist-test-slave-w6c0 krb5kdc[12264](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790584817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:17.963235 13478 webserver.cc:460] Webserver started at http://127.0.97.2:35309/ using document root <none> and password file <none>
I20260928 08:40:17.964236 13478 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:17.964397 13478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 08:40:17 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:40:18.000643 13478 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.006s sys 0.000s
I20260928 08:40:18.006749 13505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:18.008950 13478 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.001s
I20260928 08:40:18.009209 13478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "70cecfed712b46b288d01dd990d4f2b4"
format_stamp: "Formatted at 2026-09-28 08:40:14 on dist-test-slave-w6c0"
server_key: "39f24a9b3d7f2becd34eae7b5cd9e9d0"
server_key_iv: "7443341930e7592f293680c57802e045"
server_key_version: "kuduclusterkey@0"
I20260928 08:40:18.010779 13478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:18.035563 13478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:18.041294 13478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:18.041651 13478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:18.042982 13478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:40:18.044917 13478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:40:18.045040 13478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:18.045203 13478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:40:18.045315 13478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:18.112457 13478 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41059
I20260928 08:40:18.112515 13618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41059 every 8 connection(s)
I20260928 08:40:18.114523 13478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:40:18.115589 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 13478
I20260928 08:40:18.116312 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:37325/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35191
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39755
--webserver_port=37921
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:37325/kms with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:40:18 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:18.198022 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:18.124374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39755 (local address 127.0.97.2:50017)
0928 08:40:18.163269 (+ 38895us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:18.163348 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:18.166785 (+ 3437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:18.167786 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:18.167800 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:18.168475 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:18.169348 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:18.169375 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:18.171213 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:18.171222 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:18.173632 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:18.173650 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:18.175021 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:18.175029 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:18.175325 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:18.176163 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:18.176202 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:18.182917 (+ 6715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:18.193088 (+ 10171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:18.193099 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:18.193120 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:18.193849 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:18.194351 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:18.194356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:18.194360 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:18.194668 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:18.195428 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:18.195438 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:18.195858 (+ 420us) client_negotiation.cc:769] Sending connection context
0928 08:40:18.196156 (+ 298us) client_negotiation.cc:240] Negotiation successful
0928 08:40:18.196735 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38476,"thread_start_us":38204,"threads_started":1}
I20260928 08:40:18.199216 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:18.123767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50017 (local address 127.0.97.62:39755)
0928 08:40:18.155256 (+ 31489us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:18.155265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:18.163508 (+ 8243us) server_negotiation.cc:407] Connection header received
0928 08:40:18.167206 (+ 3698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:18.167215 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:18.167329 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:18.167464 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:18.169582 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:18.170974 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:18.173836 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:18.174548 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:18.174687 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:18.183343 (+ 8656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:18.183351 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:18.183357 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:18.183409 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:18.192837 (+ 9428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:18.194028 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:18.194034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:18.194038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:18.194204 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:18.194817 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:18.194821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:18.194825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:18.195190 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:18.195331 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:18.198580 (+ 3249us) server_negotiation.cc:299] Negotiation successful
0928 08:40:18.198852 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31314,"thread_start_us":31489,"threads_started":1}
I20260928 08:40:18.201406 13619 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39755
I20260928 08:40:18.201750 13619 heartbeater.cc:464] Registering TS with master...
I20260928 08:40:18.202659 13619 heartbeater.cc:511] Master 127.0.97.62:39755 requested a full tablet report, sending...
I20260928 08:40:18.204900 13276 ts_manager.cc:194] Registered new tserver with Master: 70cecfed712b46b288d01dd990d4f2b4 (127.0.97.2:41059)
I20260928 08:40:18.207244 13276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50017
Sep 28 08:40:18 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:40:18.476814 13474 heartbeater.cc:464] Registering TS with master...
I20260928 08:40:18.477308 13474 heartbeater.cc:511] Master 127.0.97.62:39755 requested a full tablet report, sending...
I20260928 08:40:18.478720 13276 ts_manager.cc:194] Registered new tserver with Master: b2a9685e507044d88776f70d802886cf (127.0.97.1:43431)
I20260928 08:40:18.479902 13276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33993
W20260928 08:40:18.636934 13620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:18.638098 13620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:18.638484 13620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:18.650942 13620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:18.651186 13620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:18.651448 13620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:40:18.667621 13620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:37325/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37921
--tserver_master_addrs=127.0.97.62:39755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:18.669623 13620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:18.671458 13620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:18.691536 13629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:18.695431 13628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:18.699707 13631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:18.700049 13620 server_base.cc:1034] running on GCE node
I20260928 08:40:18.700953 13620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:18.702808 13620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:18.704077 13620 hybrid_clock.cc:648] HybridClock initialized: now 1790584818704005 us; error 63 us; skew 500 ppm
I20260928 08:40:18.704417 13620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:18 dist-test-slave-w6c0 krb5kdc[12264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:18.708309 13620 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:40:18.711308 13620 webserver.cc:460] Webserver started at http://127.0.97.3:37921/ using document root <none> and password file <none>
I20260928 08:40:18.712248 13620 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:18.712414 13620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 08:40:18 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:40:18.749243 13620 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.007s sys 0.004s
I20260928 08:40:18.761806 13640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:18.764011 13620 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.004s
I20260928 08:40:18.764240 13620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "a33a09985dbe4cb6b8388c5ff38aad0e"
format_stamp: "Formatted at 2026-09-28 08:40:15 on dist-test-slave-w6c0"
server_key: "d9516da259dcf201bb5dabae5e5f0c5e"
server_key_iv: "2db037f1111d7c248a2998482b4e19dc"
server_key_version: "kuduclusterkey@0"
I20260928 08:40:18.765689 13620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 28 08:40:18 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:40:18.784904 13620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:18.796727 13620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:18.797117 13620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:18.802069 13620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:40:18.804015 13620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:40:18.804176 13620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 08:40:18.804343 13620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:40:18.804455 13620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:19.009148 13620 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35191
I20260928 08:40:19.009178 13755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35191 every 8 connection(s)
I20260928 08:40:19.011093 13620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:40:19.016404 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 13620
Sep 28 08:40:19 dist-test-slave-w6c0 krb5kdc[12264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:19.034847 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.017726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.0.97.62:39755)
0928 08:40:19.018927 (+ 1201us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:19.018932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:19.018949 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:40:19.018994 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:19.018998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:19.019071 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:19.019189 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:19.021224 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.022442 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.024418 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.024950 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.025088 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:19.028636 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:19.028644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:19.028650 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:19.028709 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:19.032024 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:19.032722 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:19.032729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:19.032732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:19.032874 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:19.033256 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:19.033262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:19.033265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:19.033537 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:19.033836 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:19.034159 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 08:40:19.034387 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":1097,"spinlock_wait_cycles":8448,"thread_start_us":957,"threads_started":1}
I20260928 08:40:19.037225 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.014624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39755 (local address 127.0.97.3:41433)
0928 08:40:19.015366 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:19.015402 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:19.017751 (+ 2349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:19.018903 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:19.018914 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:19.019529 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:19.020097 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:19.020115 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.021173 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.021179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:19.022995 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:19.023006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.023799 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.023816 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:19.024040 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:19.024768 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:19.024814 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:19.027631 (+ 2817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:19.032234 (+ 4603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:19.032243 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:19.032263 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:19.032887 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:19.033345 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:19.033350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:19.033354 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:19.033628 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:19.034351 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:19.034359 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:19.034874 (+ 515us) client_negotiation.cc:769] Sending connection context
0928 08:40:19.035197 (+ 323us) client_negotiation.cc:240] Negotiation successful
0928 08:40:19.035805 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":317,"threads_started":1}
I20260928 08:40:19.038223 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.014132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41433 (local address 127.0.97.62:39755)
0928 08:40:19.014651 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:19.014656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:19.015524 (+ 868us) server_negotiation.cc:407] Connection header received
0928 08:40:19.018111 (+ 2587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:19.018117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:19.018225 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:19.018306 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:19.020272 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.021007 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.023133 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.023555 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.023683 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:19.027909 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:19.027914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:19.027917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:19.027958 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:19.032053 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:19.033048 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:19.033054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:19.033058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:19.033207 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:19.033790 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:19.033796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:19.033799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:19.034139 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:19.034277 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:19.037588 (+ 3311us) server_negotiation.cc:299] Negotiation successful
0928 08:40:19.037835 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":390,"thread_start_us":264,"threads_started":1}
I20260928 08:40:19.039897 13760 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39755
I20260928 08:40:19.040227 13760 heartbeater.cc:464] Registering TS with master...
I20260928 08:40:19.040979 13760 heartbeater.cc:511] Master 127.0.97.62:39755 requested a full tablet report, sending...
I20260928 08:40:19.042578 13276 ts_manager.cc:194] Registered new tserver with Master: a33a09985dbe4cb6b8388c5ff38aad0e (127.0.97.3:35191)
I20260928 08:40:19.043756 13276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41433
I20260928 08:40:19.049810 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:40:19.069645 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.053202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37754 (local address 127.0.97.62:39755)
0928 08:40:19.053401 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:19.053408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:19.053872 (+ 464us) server_negotiation.cc:407] Connection header received
0928 08:40:19.054019 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:19.054025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:19.054119 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:19.054230 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:19.055767 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.056888 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.058207 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.058905 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.059042 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:19.062282 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:19.062289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:19.062295 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:19.062351 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:19.066559 (+ 4208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:19.067474 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:19.067480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:19.067485 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:19.067651 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:19.068045 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:19.068051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:19.068055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:19.068443 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:19.068700 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:19.069013 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 08:40:19.069295 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 08:40:19.075493 13276 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:40:19.083382 13276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:40:19.139137 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.098688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.0.97.2:41059)
0928 08:40:19.129604 (+ 30916us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:19.129613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:19.129649 (+ 36us) server_negotiation.cc:407] Connection header received
0928 08:40:19.129745 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:19.129756 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:19.130036 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:19.130241 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:19.131415 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.133068 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.136569 (+ 3501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.137518 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.137667 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:19.137920 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:19.138623 (+ 703us) server_negotiation.cc:299] Negotiation successful
0928 08:40:19.138811 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30688,"thread_start_us":30510,"threads_started":1}
I20260928 08:40:19.139171 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.098779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43431 (local address 127.0.0.1:60666)
0928 08:40:19.129805 (+ 31026us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:19.129823 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:19.129944 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:19.130477 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:19.130481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:19.130524 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:19.131151 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:19.131164 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.133211 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.133218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:19.136017 (+ 2799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:19.136030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.138153 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.138160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:19.138374 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:19.138451 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 08:40:19.138604 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 08:40:19.138738 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30924,"mutex_wait_us":376,"thread_start_us":30783,"threads_started":1}
I20260928 08:40:19.139756 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.098255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41059 (local address 127.0.0.1:41250)
0928 08:40:19.126800 (+ 28545us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:19.126855 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:19.127074 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:19.130607 (+ 3533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:19.130614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:19.130646 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:19.131163 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:19.131174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.133263 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.133272 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:19.136389 (+ 3117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:19.136403 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.137740 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.137746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:19.137956 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:19.138011 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 08:40:19.138159 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 08:40:19.138393 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28247,"mutex_wait_us":142,"thread_start_us":28055,"threads_started":1}
I20260928 08:40:19.139926 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.099333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.0.97.1:43431)
0928 08:40:19.128525 (+ 29192us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:19.128537 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:19.129947 (+ 1410us) server_negotiation.cc:407] Connection header received
0928 08:40:19.130062 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:19.130073 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:19.130270 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:19.130473 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:19.131346 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.133055 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.136170 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.137620 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.137754 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:19.137979 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:19.139441 (+ 1462us) server_negotiation.cc:299] Negotiation successful
0928 08:40:19.139611 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28979,"thread_start_us":28802,"threads_started":1}
I20260928 08:40:19.140206 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.099803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42878 (local address 127.0.97.3:35191)
0928 08:40:19.127991 (+ 28188us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:19.128000 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:19.129239 (+ 1239us) server_negotiation.cc:407] Connection header received
0928 08:40:19.129584 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:19.129594 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:19.129893 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:19.130068 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:19.132028 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.133122 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.136508 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.137933 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.138087 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:19.138292 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:19.138876 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 08:40:19.139036 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27977,"thread_start_us":27835,"threads_started":1}
I20260928 08:40:19.140607 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.098488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35191 (local address 127.0.0.1:42878)
0928 08:40:19.129093 (+ 30605us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:19.129118 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:19.129357 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:19.130147 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:19.130154 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:19.130186 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:19.131799 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:19.131808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.133263 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.133268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:19.136333 (+ 3065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:19.136353 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.138193 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.138199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:19.138395 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:19.138423 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 08:40:19.138529 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 08:40:19.138655 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30434,"mutex_wait_us":285,"thread_start_us":30229,"threads_started":1}
I20260928 08:40:19.143286 13554 tablet_service.cc:1467] Processing CreateTablet for tablet e876bb7cff2442c899342ae4687d149f (DEFAULT_TABLE table=test-table [id=eae68061552947b89a00de3dfbe4eeb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:40:19.143355 13409 tablet_service.cc:1467] Processing CreateTablet for tablet e876bb7cff2442c899342ae4687d149f (DEFAULT_TABLE table=test-table [id=eae68061552947b89a00de3dfbe4eeb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:40:19.144790 13554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e876bb7cff2442c899342ae4687d149f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:40:19.144817 13409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e876bb7cff2442c899342ae4687d149f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:40:19.145792 13688 tablet_service.cc:1467] Processing CreateTablet for tablet e876bb7cff2442c899342ae4687d149f (DEFAULT_TABLE table=test-table [id=eae68061552947b89a00de3dfbe4eeb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:40:19.147161 13688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e876bb7cff2442c899342ae4687d149f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:40:19.158072 13781 tablet_bootstrap.cc:492] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4: Bootstrap starting.
I20260928 08:40:19.159865 13782 tablet_bootstrap.cc:492] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf: Bootstrap starting.
I20260928 08:40:19.161664 13781 tablet_bootstrap.cc:654] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4: Neither blocks nor log segments found. Creating new log.
I20260928 08:40:19.162163 13783 tablet_bootstrap.cc:492] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e: Bootstrap starting.
I20260928 08:40:19.163030 13781 log.cc:824] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:19.163690 13782 tablet_bootstrap.cc:654] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf: Neither blocks nor log segments found. Creating new log.
I20260928 08:40:19.164816 13782 log.cc:824] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:19.165634 13783 tablet_bootstrap.cc:654] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e: Neither blocks nor log segments found. Creating new log.
I20260928 08:40:19.166385 13781 tablet_bootstrap.cc:492] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4: No bootstrap required, opened a new log
I20260928 08:40:19.166729 13781 ts_tablet_manager.cc:1397] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 08:40:19.166929 13783 log.cc:824] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:19.168421 13782 tablet_bootstrap.cc:492] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf: No bootstrap required, opened a new log
I20260928 08:40:19.168680 13782 ts_tablet_manager.cc:1397] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 08:40:19.169410 13783 tablet_bootstrap.cc:492] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e: No bootstrap required, opened a new log
I20260928 08:40:19.169667 13783 ts_tablet_manager.cc:1397] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 08:40:19.177011 13783 raft_consensus.cc:348] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cecfed712b46b288d01dd990d4f2b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41059 } } peers { permanent_uuid: "a33a09985dbe4cb6b8388c5ff38aad0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35191 } } peers { permanent_uuid: "b2a9685e507044d88776f70d802886cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } }
I20260928 08:40:19.177379 13781 raft_consensus.cc:348] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cecfed712b46b288d01dd990d4f2b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41059 } } peers { permanent_uuid: "a33a09985dbe4cb6b8388c5ff38aad0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35191 } } peers { permanent_uuid: "b2a9685e507044d88776f70d802886cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } }
I20260928 08:40:19.177663 13783 raft_consensus.cc:374] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:40:19.177775 13783 raft_consensus.cc:729] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a33a09985dbe4cb6b8388c5ff38aad0e, State: Initialized, Role: FOLLOWER
I20260928 08:40:19.177893 13781 raft_consensus.cc:374] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:40:19.178052 13781 raft_consensus.cc:729] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70cecfed712b46b288d01dd990d4f2b4, State: Initialized, Role: FOLLOWER
I20260928 08:40:19.178596 13783 consensus_queue.cc:260] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cecfed712b46b288d01dd990d4f2b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41059 } } peers { permanent_uuid: "a33a09985dbe4cb6b8388c5ff38aad0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35191 } } peers { permanent_uuid: "b2a9685e507044d88776f70d802886cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } }
I20260928 08:40:19.178707 13781 consensus_queue.cc:260] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cecfed712b46b288d01dd990d4f2b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41059 } } peers { permanent_uuid: "a33a09985dbe4cb6b8388c5ff38aad0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35191 } } peers { permanent_uuid: "b2a9685e507044d88776f70d802886cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } }
I20260928 08:40:19.179723 13760 heartbeater.cc:503] Master 127.0.97.62:39755 was elected leader, sending a full tablet report...
I20260928 08:40:19.179718 13782 raft_consensus.cc:348] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cecfed712b46b288d01dd990d4f2b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41059 } } peers { permanent_uuid: "a33a09985dbe4cb6b8388c5ff38aad0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35191 } } peers { permanent_uuid: "b2a9685e507044d88776f70d802886cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } }
I20260928 08:40:19.180243 13782 raft_consensus.cc:374] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:40:19.180349 13782 raft_consensus.cc:729] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a9685e507044d88776f70d802886cf, State: Initialized, Role: FOLLOWER
I20260928 08:40:19.180457 13783 ts_tablet_manager.cc:1428] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260928 08:40:19.181167 13782 consensus_queue.cc:260] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cecfed712b46b288d01dd990d4f2b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41059 } } peers { permanent_uuid: "a33a09985dbe4cb6b8388c5ff38aad0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35191 } } peers { permanent_uuid: "b2a9685e507044d88776f70d802886cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } }
I20260928 08:40:19.182372 13474 heartbeater.cc:503] Master 127.0.97.62:39755 was elected leader, sending a full tablet report...
I20260928 08:40:19.183318 13782 ts_tablet_manager.cc:1428] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260928 08:40:19.184273 13619 heartbeater.cc:503] Master 127.0.97.62:39755 was elected leader, sending a full tablet report...
I20260928 08:40:19.185063 13781 ts_tablet_manager.cc:1428] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
W20260928 08:40:19.186084 13475 tablet.cc:2369] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:40:19.262964 13761 tablet.cc:2369] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:40:19.286608 13789 raft_consensus.cc:482] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:40:19.286924 13789 raft_consensus.cc:504] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cecfed712b46b288d01dd990d4f2b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41059 } } peers { permanent_uuid: "a33a09985dbe4cb6b8388c5ff38aad0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35191 } } peers { permanent_uuid: "b2a9685e507044d88776f70d802886cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } }
I20260928 08:40:19.288761 13789 leader_election.cc:288] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70cecfed712b46b288d01dd990d4f2b4 (127.0.97.2:41059), a33a09985dbe4cb6b8388c5ff38aad0e (127.0.97.3:35191)
I20260928 08:40:19.295747 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.289111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53473 (local address 127.0.97.2:41059)
0928 08:40:19.289341 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:19.289351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:19.290182 (+ 831us) server_negotiation.cc:407] Connection header received
0928 08:40:19.290379 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:19.290383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:19.290455 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:19.290610 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:19.291241 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.291950 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.293763 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.294723 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.294867 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:19.294990 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:19.295352 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 08:40:19.295478 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:40:19.295895 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.289336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39751 (local address 127.0.97.3:35191)
0928 08:40:19.289557 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:19.289566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:19.290028 (+ 462us) server_negotiation.cc:407] Connection header received
0928 08:40:19.290259 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:19.290266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:19.290343 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:19.290452 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:19.291101 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.291815 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.293865 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.294888 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.295039 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:19.295146 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:19.295502 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 08:40:19.295622 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 08:40:19.295921 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.289160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41059 (local address 127.0.97.1:53473)
0928 08:40:19.290079 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:19.290094 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:19.290242 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:19.290634 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:19.290638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:19.290677 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:19.291091 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:19.291099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.292094 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.292103 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:19.293602 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:19.293615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.294939 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.294945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:19.295144 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:19.295184 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 08:40:19.295327 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 08:40:19.295477 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":21,"thread_start_us":601,"threads_started":1}
I20260928 08:40:19.296375 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.289121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35191 (local address 127.0.97.1:39751)
0928 08:40:19.289903 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:19.289921 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:19.290103 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:19.290468 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:19.290473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:19.290497 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:19.290947 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:19.290956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.291964 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.291970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:19.293713 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:19.293725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.295133 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.295140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:19.295350 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:19.295374 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 08:40:19.295477 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 08:40:19.295743 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":37,"thread_start_us":434,"threads_started":1}
I20260928 08:40:19.297083 13574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e876bb7cff2442c899342ae4687d149f" candidate_uuid: "b2a9685e507044d88776f70d802886cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70cecfed712b46b288d01dd990d4f2b4" is_pre_election: true
I20260928 08:40:19.297382 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e876bb7cff2442c899342ae4687d149f" candidate_uuid: "b2a9685e507044d88776f70d802886cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a33a09985dbe4cb6b8388c5ff38aad0e" is_pre_election: true
I20260928 08:40:19.297722 13574 raft_consensus.cc:2454] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a9685e507044d88776f70d802886cf in term 0.
I20260928 08:40:19.297821 13708 raft_consensus.cc:2454] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a9685e507044d88776f70d802886cf in term 0.
I20260928 08:40:19.298487 13363 leader_election.cc:302] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 70cecfed712b46b288d01dd990d4f2b4, a33a09985dbe4cb6b8388c5ff38aad0e, b2a9685e507044d88776f70d802886cf; no voters:
I20260928 08:40:19.298957 13789 raft_consensus.cc:2781] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:40:19.299111 13789 raft_consensus.cc:482] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:40:19.299229 13789 raft_consensus.cc:3037] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:40:19.301383 13789 raft_consensus.cc:504] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cecfed712b46b288d01dd990d4f2b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41059 } } peers { permanent_uuid: "a33a09985dbe4cb6b8388c5ff38aad0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35191 } } peers { permanent_uuid: "b2a9685e507044d88776f70d802886cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } }
I20260928 08:40:19.301996 13789 leader_election.cc:288] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [CANDIDATE]: Term 1 election: Requested vote from peers 70cecfed712b46b288d01dd990d4f2b4 (127.0.97.2:41059), a33a09985dbe4cb6b8388c5ff38aad0e (127.0.97.3:35191)
I20260928 08:40:19.302575 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e876bb7cff2442c899342ae4687d149f" candidate_uuid: "b2a9685e507044d88776f70d802886cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a33a09985dbe4cb6b8388c5ff38aad0e"
I20260928 08:40:19.302593 13574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e876bb7cff2442c899342ae4687d149f" candidate_uuid: "b2a9685e507044d88776f70d802886cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70cecfed712b46b288d01dd990d4f2b4"
I20260928 08:40:19.302790 13708 raft_consensus.cc:3037] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:40:19.302802 13574 raft_consensus.cc:3037] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:40:19.304936 13708 raft_consensus.cc:2454] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a9685e507044d88776f70d802886cf in term 1.
I20260928 08:40:19.304955 13574 raft_consensus.cc:2454] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a9685e507044d88776f70d802886cf in term 1.
I20260928 08:40:19.305367 13362 leader_election.cc:302] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a33a09985dbe4cb6b8388c5ff38aad0e, b2a9685e507044d88776f70d802886cf; no voters:
I20260928 08:40:19.305670 13789 raft_consensus.cc:2781] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:40:19.306113 13789 raft_consensus.cc:686] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [term 1 LEADER]: Becoming Leader. State: Replica: b2a9685e507044d88776f70d802886cf, State: Running, Role: LEADER
I20260928 08:40:19.306644 13789 consensus_queue.cc:237] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cecfed712b46b288d01dd990d4f2b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41059 } } peers { permanent_uuid: "a33a09985dbe4cb6b8388c5ff38aad0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35191 } } peers { permanent_uuid: "b2a9685e507044d88776f70d802886cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } }
I20260928 08:40:19.311715 13275 catalog_manager.cc:5495] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf reported cstate change: term changed from 0 to 1, leader changed from <none> to b2a9685e507044d88776f70d802886cf (127.0.97.1). New cstate: current_term: 1 leader_uuid: "b2a9685e507044d88776f70d802886cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cecfed712b46b288d01dd990d4f2b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a33a09985dbe4cb6b8388c5ff38aad0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a9685e507044d88776f70d802886cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } health_report { overall_health: HEALTHY } } }
I20260928 08:40:19.331133 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.325769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60672 (local address 127.0.97.1:43431)
0928 08:40:19.325986 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:19.325994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:19.326016 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:40:19.326118 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:19.326121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:19.326212 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:19.326362 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:40:19.326790 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.327529 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.328928 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.329326 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.329460 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:19.329948 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:40:19.330465 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:40:19.330684 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:19.330767 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 08:40:19.330852 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 08:40:19.339888 13708 raft_consensus.cc:1261] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e [term 1 FOLLOWER]: Refusing update from remote peer b2a9685e507044d88776f70d802886cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:40:19.339884 13574 raft_consensus.cc:1261] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4 [term 1 FOLLOWER]: Refusing update from remote peer b2a9685e507044d88776f70d802886cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:40:19.340893 13792 consensus_queue.cc:1035] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "70cecfed712b46b288d01dd990d4f2b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:40:19.341221 13789 consensus_queue.cc:1035] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a33a09985dbe4cb6b8388c5ff38aad0e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:40:19.353654 13796 mvcc.cc:205] Tried to move back new op lower bound from 7334235420005838848 to 7334235419883786240. Current Snapshot: MvccSnapshot[applied={T|T < 7334235420005838848}]
I20260928 08:40:19.372920 13275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:40:19.373245 13275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:40:19.377061 13275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0bfbc72e1d14f65b9a5c62ab8a52ff6: Sending DeleteTablet for 3 replicas of tablet e876bb7cff2442c899342ae4687d149f
I20260928 08:40:19.378263 13554 tablet_service.cc:1514] Processing DeleteTablet for tablet e876bb7cff2442c899342ae4687d149f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:40:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41250
I20260928 08:40:19.378255 13688 tablet_service.cc:1514] Processing DeleteTablet for tablet e876bb7cff2442c899342ae4687d149f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:40:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42878
I20260928 08:40:19.378575 13409 tablet_service.cc:1514] Processing DeleteTablet for tablet e876bb7cff2442c899342ae4687d149f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:40:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60666
I20260928 08:40:19.386729 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 13313
I20260928 08:40:19.388398 13804 tablet_replica.cc:330] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e: stopping tablet replica
I20260928 08:40:19.388986 13804 raft_consensus.cc:2229] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:40:19.389681 13804 raft_consensus.cc:2258] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:40:19.390492 13806 tablet_replica.cc:330] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf: stopping tablet replica
I20260928 08:40:19.391108 13806 raft_consensus.cc:2229] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:40:19.391902 13806 raft_consensus.cc:2258] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:40:19.395051 13805 tablet_replica.cc:330] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4: stopping tablet replica
I20260928 08:40:19.395072 13804 ts_tablet_manager.cc:1905] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:40:19.395524 13805 raft_consensus.cc:2229] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:40:19.396073 13805 raft_consensus.cc:2258] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:40:19.398306 13805 ts_tablet_manager.cc:1905] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:40:19.404168 13806 ts_tablet_manager.cc:1905] T e876bb7cff2442c899342ae4687d149f P b2a9685e507044d88776f70d802886cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:40:19.405436 13805 ts_tablet_manager.cc:1918] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:40:19.405658 13805 log.cc:1196] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/e876bb7cff2442c899342ae4687d149f
I20260928 08:40:19.406252 13805 ts_tablet_manager.cc:1939] T e876bb7cff2442c899342ae4687d149f P 70cecfed712b46b288d01dd990d4f2b4: Deleting consensus metadata
I20260928 08:40:19.408026 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.388830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.0.97.1:43431)
0928 08:40:19.389099 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:19.389108 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:19.389131 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:40:19.389193 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:19.389198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:19.389284 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:19.389421 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:19.391658 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.392745 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.394534 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.395263 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.395530 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:19.399283 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:19.399298 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:19.399317 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:19.399395 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:19.403590 (+ 4195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:19.404553 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:19.404561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:19.404564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:19.404744 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:19.405214 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:19.405219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:19.405222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:19.405695 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:19.406001 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:19.407341 (+ 1340us) server_negotiation.cc:299] Negotiation successful
0928 08:40:19.407651 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 08:40:19.408469 13261 catalog_manager.cc:4841] TS 70cecfed712b46b288d01dd990d4f2b4 (127.0.97.2:41059): tablet e876bb7cff2442c899342ae4687d149f (table test-table [id=eae68061552947b89a00de3dfbe4eeb8]) successfully deleted
I20260928 08:40:19.408969 13469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:19.411392 13804 ts_tablet_manager.cc:1918] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:40:19.411587 13804 log.cc:1196] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/e876bb7cff2442c899342ae4687d149f
I20260928 08:40:19.412122 13804 ts_tablet_manager.cc:1939] T e876bb7cff2442c899342ae4687d149f P a33a09985dbe4cb6b8388c5ff38aad0e: Deleting consensus metadata
I20260928 08:40:19.414151 13260 catalog_manager.cc:4841] TS a33a09985dbe4cb6b8388c5ff38aad0e (127.0.97.3:35191): tablet e876bb7cff2442c899342ae4687d149f (table test-table [id=eae68061552947b89a00de3dfbe4eeb8]) successfully deleted
I20260928 08:40:19.508600 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 13313
W20260928 08:40:19.527215 13263 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43431 (error 108)
I20260928 08:40:19.527318 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 13478
I20260928 08:40:19.529093 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.528615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43431 (local address 127.0.0.1:60682)
0928 08:40:19.528800 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:19.528921 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 08:40:19.529351 13263 catalog_manager.cc:4584] TS b2a9685e507044d88776f70d802886cf (127.0.97.1:43431): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e876bb7cff2442c899342ae4687d149f: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43431: connect: Connection refused (error 111)
I20260928 08:40:19.541495 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.528021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.0.97.2:41059)
0928 08:40:19.528212 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:19.528219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:19.528370 (+ 151us) server_negotiation.cc:407] Connection header received
0928 08:40:19.528579 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:19.528583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:19.528670 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:19.528775 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:19.530240 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.531070 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.532393 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.532815 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.532925 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:19.535508 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:19.535522 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:19.535538 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:19.535576 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:19.538709 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:19.539528 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:19.539535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:19.539538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:19.539687 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:19.540071 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:19.540075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:19.540078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:19.540439 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:19.540703 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:19.540935 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 08:40:19.541108 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:40:19.542191 13614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:19.583076 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.582533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43431 (local address 127.0.0.1:60698)
0928 08:40:19.582801 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:19.582912 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 08:40:19.584180 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.583819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43431 (local address 127.0.0.1:60708)
0928 08:40:19.583988 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:19.584067 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 08:40:19.634080 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 13478
I20260928 08:40:19.644904 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.644367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43431 (local address 127.0.0.1:60714)
0928 08:40:19.644605 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:19.644709 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 08:40:19.646560 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.646009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43431 (local address 127.0.0.1:60718)
0928 08:40:19.646231 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:19.646330 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 08:40:19.652405 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 13620
I20260928 08:40:19.667989 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.653216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42894 (local address 127.0.97.3:35191)
0928 08:40:19.653382 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:19.653386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:19.653406 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:40:19.653511 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:19.653515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:19.653599 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:19.653705 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:19.655092 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.655880 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.657687 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:19.658130 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:19.658259 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:19.661233 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:19.661246 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:19.661263 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:19.661303 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:19.664827 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:19.665634 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:19.665642 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:19.665644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:19.665783 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:19.666196 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:19.666202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:19.666205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:19.666674 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:19.666980 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:19.667410 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 08:40:19.667643 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 08:40:19.668701 13750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:19.727829 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.727271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43431 (local address 127.0.0.1:60728)
0928 08:40:19.727539 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:19.727637 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260928 08:40:19.728901 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:19.728531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43431 (local address 127.0.0.1:60742)
0928 08:40:19.728718 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:19.728802 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 08:40:19.760258 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 13620
I20260928 08:40:19.778214 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 13243
I20260928 08:40:19.779562 13306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:19.896427 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 13243
I20260928 08:40:19.919936 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 08:40:20.256057 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 08:40:20.256299 388 mini_ranger.cc:66] Stopping Ranger...
I20260928 08:40:20.601830 388 mini_ranger.cc:68] Stopped Ranger
2026-09-28 08:40:20.602 UTC [12294] LOG: received smart shutdown request
2026-09-28 08:40:20.605 UTC [12294] LOG: background worker "logical replication launcher" (PID 12303) exited with exit code 1
2026-09-28 08:40:20.605 UTC [12298] LOG: shutting down
2026-09-28 08:40:20.769 UTC [12294] LOG: database system is shut down
2026-09-28T08:40:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53929 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:40:20 dist-test-slave-w6c0 krb5kdc[13820](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:40:20 dist-test-slave-w6c0 krb5kdc[13820](info): set up 2 sockets
Sep 28 08:40:20 dist-test-slave-w6c0 krb5kdc[13820](info): commencing operation
krb5kdc: starting...
W20260928 08:40:21.994807 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:40:22 dist-test-slave-w6c0 krb5kdc[13820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:40:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:40:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:40:22.217582 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41493
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41493,127.0.97.61:36877,127.0.97.60:37747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:40:22.542346 13833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:22.542654 13833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:22.542833 13833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:22.550099 13833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:40:22.550199 13833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:22.550288 13833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:22.550396 13833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:40:22.550477 13833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:40:22.560230 13833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41493,127.0.97.61:36877,127.0.97.60:37747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:22.561511 13833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:22.562872 13833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:22.574580 13841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:22.574666 13838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:22.574625 13839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:22.575266 13833 server_base.cc:1034] running on GCE node
I20260928 08:40:22.575984 13833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:22.577291 13833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:22.578588 13833 hybrid_clock.cc:648] HybridClock initialized: now 1790584822578533 us; error 53 us; skew 500 ppm
I20260928 08:40:22.578949 13833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:22 dist-test-slave-w6c0 krb5kdc[13820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:22.587766 13833 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:40:22.589322 13833 webserver.cc:460] Webserver started at http://127.0.97.62:45087/ using document root <none> and password file <none>
I20260928 08:40:22.590005 13833 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:22.590152 13833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:22.590463 13833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:40:22.593339 13833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "4b2eea2a968641c18dc3b80539f9f878"
format_stamp: "Formatted at 2026-09-28 08:40:22 on dist-test-slave-w6c0"
server_key: "0ba6ec7d4401e4c2b4545f6289786af1"
server_key_iv: "324a4076f4fc245c00a286d3da4dcce0"
server_key_version: "encryptionkey@0"
I20260928 08:40:22.594138 13833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "4b2eea2a968641c18dc3b80539f9f878"
format_stamp: "Formatted at 2026-09-28 08:40:22 on dist-test-slave-w6c0"
server_key: "0ba6ec7d4401e4c2b4545f6289786af1"
server_key_iv: "324a4076f4fc245c00a286d3da4dcce0"
server_key_version: "encryptionkey@0"
I20260928 08:40:22.599174 13833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260928 08:40:22.602391 13848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:22.603802 13833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:40:22.603991 13833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "4b2eea2a968641c18dc3b80539f9f878"
format_stamp: "Formatted at 2026-09-28 08:40:22 on dist-test-slave-w6c0"
server_key: "0ba6ec7d4401e4c2b4545f6289786af1"
server_key_iv: "324a4076f4fc245c00a286d3da4dcce0"
server_key_version: "encryptionkey@0"
I20260928 08:40:22.604198 13833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:22.622486 13833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:22.626812 13833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:22.627094 13833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:22.645560 13833 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41493
I20260928 08:40:22.645597 13899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41493 every 8 connection(s)
I20260928 08:40:22.646981 13833 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:40:22.647666 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 13833
I20260928 08:40:22.647909 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:40:22.648641 388 external_mini_cluster.cc:1428] Setting key 218cc6576e2bcee89e7e7548a35240db
I20260928 08:40:22.650914 13900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:40:22.658280 13900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 28 08:40:22 dist-test-slave-w6c0 krb5kdc[13820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:22.682960 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:22.663288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41194 (local address 127.0.97.62:41493)
0928 08:40:22.663984 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:22.663994 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:22.664046 (+ 52us) server_negotiation.cc:407] Connection header received
0928 08:40:22.665298 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:22.665308 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:22.665611 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:22.665840 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:22.667006 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:22.669028 (+ 2022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:22.670566 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:22.671114 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:22.671257 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:22.674974 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:22.674987 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:22.675005 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:22.675048 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:22.678960 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:22.679687 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:22.679693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:22.679695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:22.679836 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:22.680302 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:22.680305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:22.680307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:22.680778 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:22.680933 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:22.681577 (+ 644us) server_negotiation.cc:299] Negotiation successful
0928 08:40:22.681847 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":457,"thread_start_us":299,"threads_started":1}
I20260928 08:40:22.683054 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:22.662035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41493 (local address 127.0.0.1:41194)
0928 08:40:22.662860 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:22.662915 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:22.664914 (+ 1999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:22.665871 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:22.665876 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:22.666412 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:22.666809 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:22.666833 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:22.669191 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:22.669194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:22.670365 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:22.670383 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:22.671369 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:22.671383 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:22.671505 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:22.672313 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:22.672350 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:22.674733 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:22.679100 (+ 4367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:22.679110 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:22.679119 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:22.679510 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:22.679954 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:22.679960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:22.679965 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:22.680189 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:22.680931 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:22.680943 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:22.681197 (+ 254us) client_negotiation.cc:769] Sending connection context
0928 08:40:22.681465 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 08:40:22.682016 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":50,"thread_start_us":343,"threads_started":1}
I20260928 08:40:22.686097 13900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:40:22.687278 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:22.686922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36877 (local address 127.0.0.1:49378)
0928 08:40:22.687082 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:22.687180 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 08:40:22.687670 13852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36877: connect: Connection refused (error 111)
I20260928 08:40:22.689188 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:22.688843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36877 (local address 127.0.0.1:49380)
0928 08:40:22.688995 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:22.689068 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260928 08:40:22.689491 13900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36877: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36877: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260928 08:40:22.702208 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36877
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41493,127.0.97.61:36877,127.0.97.60:37747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:40:22.738899 13900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } attempt: 1
I20260928 08:40:22.740106 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:22.739677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36877 (local address 127.0.0.1:49392)
0928 08:40:22.739877 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:22.739979 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 08:40:22.741061 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:22.740728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36877 (local address 127.0.0.1:49402)
0928 08:40:22.740874 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:22.740945 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 08:40:22.741240 13900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36877: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36877: connect: Connection refused (error 111)
I20260928 08:40:22.809463 13900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } attempt: 2
I20260928 08:40:22.810665 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:22.810125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36877 (local address 127.0.0.1:49412)
0928 08:40:22.810384 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:22.810470 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 08:40:22.811679 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:22.811300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36877 (local address 127.0.0.1:49428)
0928 08:40:22.811486 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:22.811557 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260928 08:40:22.811846 13900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36877: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36877: connect: Connection refused (error 111)
I20260928 08:40:22.903168 13900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } attempt: 3
I20260928 08:40:22.904383 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:22.903952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36877 (local address 127.0.0.1:49434)
0928 08:40:22.904149 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:22.904238 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 08:40:22.905398 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:22.905022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36877 (local address 127.0.0.1:49442)
0928 08:40:22.905180 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:22.905255 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 08:40:22.905563 13900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36877: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36877: connect: Connection refused (error 111)
W20260928 08:40:23.022251 13907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:23.022612 13907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:23.022770 13907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:23.030027 13907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:40:23.030179 13907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:23.030292 13907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:23.030390 13907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:40:23.030476 13907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:40:23.040854 13907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41493,127.0.97.61:36877,127.0.97.60:37747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:23.042282 13907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:23.043602 13907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:40:23.048803 13900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } attempt: 4
I20260928 08:40:23.050201 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.049565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36877 (local address 127.0.0.1:49448)
0928 08:40:23.049995 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.050076 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286}
I20260928 08:40:23.051182 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.050838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36877 (local address 127.0.0.1:49452)
0928 08:40:23.050983 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.051072 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 08:40:23.051366 13900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36877: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36877: connect: Connection refused (error 111)
W20260928 08:40:23.055789 13915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:23.055881 13912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:23.055855 13913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:23.056704 13907 server_base.cc:1034] running on GCE node
I20260928 08:40:23.057523 13907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:23.058982 13907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:23.060216 13907 hybrid_clock.cc:648] HybridClock initialized: now 1790584823060179 us; error 48 us; skew 500 ppm
I20260928 08:40:23.060560 13907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:23 dist-test-slave-w6c0 krb5kdc[13820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:23.063675 13907 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260928 08:40:23.065042 13907 webserver.cc:460] Webserver started at http://127.0.97.61:37907/ using document root <none> and password file <none>
I20260928 08:40:23.065686 13907 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:23.065783 13907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:23.066038 13907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:40:23.069015 13907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-1/data/instance:
uuid: "3e244e01425647c4bb93fc1286232581"
format_stamp: "Formatted at 2026-09-28 08:40:23 on dist-test-slave-w6c0"
server_key: "1322b900ff859e25a9e6d91a66c3e93d"
server_key_iv: "41d9e1249ef8092b9a6fd41f79e7303d"
server_key_version: "encryptionkey@0"
I20260928 08:40:23.069738 13907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-1/wal/instance:
uuid: "3e244e01425647c4bb93fc1286232581"
format_stamp: "Formatted at 2026-09-28 08:40:23 on dist-test-slave-w6c0"
server_key: "1322b900ff859e25a9e6d91a66c3e93d"
server_key_iv: "41d9e1249ef8092b9a6fd41f79e7303d"
server_key_version: "encryptionkey@0"
I20260928 08:40:23.074839 13907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260928 08:40:23.078194 13922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:23.079567 13907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:40:23.079712 13907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-1/wal
uuid: "3e244e01425647c4bb93fc1286232581"
format_stamp: "Formatted at 2026-09-28 08:40:23 on dist-test-slave-w6c0"
server_key: "1322b900ff859e25a9e6d91a66c3e93d"
server_key_iv: "41d9e1249ef8092b9a6fd41f79e7303d"
server_key_version: "encryptionkey@0"
I20260928 08:40:23.079879 13907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:23.102079 13907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:23.106150 13907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:23.106390 13907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:23.125663 13907 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36877
I20260928 08:40:23.125696 13973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36877 every 8 connection(s)
I20260928 08:40:23.127175 13907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-1/data/info.pb
I20260928 08:40:23.130893 13974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:40:23.134238 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 13907
I20260928 08:40:23.134450 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-1/wal/instance
I20260928 08:40:23.134883 388 external_mini_cluster.cc:1428] Setting key 3908932ad5afb40f83ccf3304ce9c317
I20260928 08:40:23.137682 13974 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:40:23 dist-test-slave-w6c0 krb5kdc[13820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260928 08:40:23.161180 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.140481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.0.97.62:41493)
0928 08:40:23.140709 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:23.140717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:23.141886 (+ 1169us) server_negotiation.cc:407] Connection header received
0928 08:40:23.144564 (+ 2678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:23.144569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:23.144639 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:23.144734 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:23.146800 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.147469 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.149960 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.150374 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.150488 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.154880 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:23.154885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:23.154888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:23.154927 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:23.157703 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.158553 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.158563 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.158565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.158787 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.159211 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.159215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.159216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.159543 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:23.159701 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:23.160439 (+ 738us) server_negotiation.cc:299] Negotiation successful
0928 08:40:23.160714 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:40:23.161803 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.140886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41493 (local address 127.0.0.1:41198)
0928 08:40:23.141705 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.141755 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:23.144260 (+ 2505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:23.144995 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:23.145008 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:23.145796 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:23.146620 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:23.146646 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.147631 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.147639 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:23.149729 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:23.149746 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.150642 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.150651 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:23.150948 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.151711 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:23.151750 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:23.154657 (+ 2907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:23.157950 (+ 3293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:23.157957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:23.157969 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:23.158398 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:23.158924 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:23.158928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:23.158931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:23.159103 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:23.159697 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:23.159703 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:23.160051 (+ 348us) client_negotiation.cc:769] Sending connection context
0928 08:40:23.160292 (+ 241us) client_negotiation.cc:240] Negotiation successful
0928 08:40:23.160662 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":358,"threads_started":1}
I20260928 08:40:23.163571 13974 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:40:23 dist-test-slave-w6c0 krb5kdc[13820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 08:40:23.179220 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.164382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36877 (local address 127.0.0.1:49456)
0928 08:40:23.164547 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.164561 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:23.164641 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:23.165682 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:23.165686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:23.166121 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:23.166289 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:23.166303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.167687 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.167695 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:23.169266 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:23.169282 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.170086 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.170098 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:23.170365 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.170979 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:23.171005 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:23.173510 (+ 2505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:23.176770 (+ 3260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:23.176775 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:23.176779 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:23.177135 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:23.177485 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:23.177492 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:23.177495 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:23.177633 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:23.178207 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:23.178213 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:23.178375 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 08:40:23.178547 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 08:40:23.178824 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 08:40:23.179324 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.164679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.0.97.61:36877)
0928 08:40:23.165292 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:23.165298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:23.165323 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:40:23.165363 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:23.165369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:23.165530 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:23.165689 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:23.166462 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.167524 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.169413 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.169929 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.170022 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.173707 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:23.173717 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:23.173730 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:23.173759 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:23.176627 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.177262 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.177270 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.177273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.177383 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.177731 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.177739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.177742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.178086 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:23.178310 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:23.178681 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 08:40:23.178944 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":478,"thread_start_us":356,"threads_started":1}
I20260928 08:40:23.181705 13974 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:40:23.182956 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.182467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58274)
0928 08:40:23.182680 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.182808 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260928 08:40:23.183264 13924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
I20260928 08:40:23.184971 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.184570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58286)
0928 08:40:23.184762 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.184854 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260928 08:40:23.185213 13974 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37747: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260928 08:40:23.189087 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37747
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41493,127.0.97.61:36877,127.0.97.60:37747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:40:23.234678 13974 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } attempt: 1
I20260928 08:40:23.235924 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.235483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58290)
0928 08:40:23.235676 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.235777 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 08:40:23.236999 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.236547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58300)
0928 08:40:23.236749 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.236864 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260928 08:40:23.237216 13974 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37747: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
I20260928 08:40:23.305505 13974 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } attempt: 2
I20260928 08:40:23.306769 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.306198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58304)
0928 08:40:23.306421 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.306580 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 08:40:23.307778 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.307420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58314)
0928 08:40:23.307567 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.307647 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 08:40:23.308023 13974 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37747: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
I20260928 08:40:23.350683 13900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } attempt: 5
Sep 28 08:40:23 dist-test-slave-w6c0 krb5kdc[13820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 08:40:23.367309 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.351445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36877 (local address 127.0.0.1:49466)
0928 08:40:23.351673 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.351695 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:23.351828 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:23.352190 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:23.352195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:23.352719 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:23.352944 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:23.352955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.354654 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.354659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:23.356024 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:23.356039 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.356846 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.356852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:23.357232 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.358411 (+ 1179us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:23.358460 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:23.360975 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:23.365136 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:23.365140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:23.365143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:23.365494 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:23.365874 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:23.365877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:23.365879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:23.366017 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:23.366625 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:23.366628 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:23.366754 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 08:40:23.366865 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 08:40:23.367072 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 08:40:23.367528 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.351703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49466 (local address 127.0.97.61:36877)
0928 08:40:23.351893 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:23.351898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:23.351917 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:40:23.351973 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:23.351977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:23.352038 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:23.352121 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:23.353268 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.354452 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.356220 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.356666 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.356770 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.361209 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:23.361219 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:23.361226 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:23.361277 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:23.364982 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.365612 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.365618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.365622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.365765 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.366118 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.366123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.366126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.366487 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:23.366660 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:23.366999 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 08:40:23.367219 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:40:23.368542 13900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:40:23.369668 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.369169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58330)
0928 08:40:23.369399 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.369493 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 08:40:23.370836 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.370365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58346)
0928 08:40:23.370567 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.370683 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 08:40:23.371093 13900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37747: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
I20260928 08:40:23.399276 13974 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } attempt: 3
I20260928 08:40:23.400701 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.400073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58350)
0928 08:40:23.400365 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.400493 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 08:40:23.401867 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.401381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58366)
0928 08:40:23.401623 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.401723 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260928 08:40:23.402107 13974 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37747: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
I20260928 08:40:23.422333 13900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } attempt: 1
I20260928 08:40:23.423593 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.423005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58370)
0928 08:40:23.423256 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.423376 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 08:40:23.424641 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.424244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58380)
0928 08:40:23.424416 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.424508 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 08:40:23.424844 13900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37747: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
I20260928 08:40:23.493120 13900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } attempt: 2
I20260928 08:40:23.494431 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.493835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58392)
0928 08:40:23.494107 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.494233 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260928 08:40:23.495467 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.495097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58406)
0928 08:40:23.495248 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.495332 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 08:40:23.495744 13900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37747: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
W20260928 08:40:23.520778 13981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:23.521134 13981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:23.521287 13981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:23.528463 13981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:40:23.528613 13981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:23.528723 13981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:23.528824 13981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:40:23.528916 13981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:40:23.538702 13981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41493,127.0.97.61:36877,127.0.97.60:37747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:23.540089 13981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:23.541438 13981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:40:23.545339 13974 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } attempt: 4
I20260928 08:40:23.546597 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.545983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58410)
0928 08:40:23.546279 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.546377 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 08:40:23.547839 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.547373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58424)
0928 08:40:23.547605 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.547701 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260928 08:40:23.548285 13974 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37747: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
W20260928 08:40:23.553642 13986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:23.553666 13987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:23.553603 13989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:23.554384 13981 server_base.cc:1034] running on GCE node
I20260928 08:40:23.555190 13981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:23.556555 13981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:23.557782 13981 hybrid_clock.cc:648] HybridClock initialized: now 1790584823557740 us; error 66 us; skew 500 ppm
I20260928 08:40:23.558128 13981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:23 dist-test-slave-w6c0 krb5kdc[13820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:23.561122 13981 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260928 08:40:23.562664 13981 webserver.cc:460] Webserver started at http://127.0.97.60:36689/ using document root <none> and password file <none>
I20260928 08:40:23.563355 13981 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:23.563499 13981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:23.563798 13981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:40:23.566778 13981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-2/data/instance:
uuid: "a63dad4209a54b08953a8dcf389d428a"
format_stamp: "Formatted at 2026-09-28 08:40:23 on dist-test-slave-w6c0"
server_key: "d13d07583657f568f3bc070f44b67f9e"
server_key_iv: "04e89e1b0dda10420a43c285a5bfba17"
server_key_version: "encryptionkey@0"
I20260928 08:40:23.567574 13981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-2/wal/instance:
uuid: "a63dad4209a54b08953a8dcf389d428a"
format_stamp: "Formatted at 2026-09-28 08:40:23 on dist-test-slave-w6c0"
server_key: "d13d07583657f568f3bc070f44b67f9e"
server_key_iv: "04e89e1b0dda10420a43c285a5bfba17"
server_key_version: "encryptionkey@0"
I20260928 08:40:23.572549 13981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:40:23.575798 13996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:23.577039 13981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:40:23.577214 13981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-2/wal
uuid: "a63dad4209a54b08953a8dcf389d428a"
format_stamp: "Formatted at 2026-09-28 08:40:23 on dist-test-slave-w6c0"
server_key: "d13d07583657f568f3bc070f44b67f9e"
server_key_iv: "04e89e1b0dda10420a43c285a5bfba17"
server_key_version: "encryptionkey@0"
I20260928 08:40:23.577420 13981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:23.593466 13981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:23.597323 13981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:23.597604 13981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:23.601944 13900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } attempt: 3
I20260928 08:40:23.603295 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.602630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58428)
0928 08:40:23.602942 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.603059 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 08:40:23.604494 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.604024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58436)
0928 08:40:23.604236 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.604343 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260928 08:40:23.604701 13900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:37747: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37747: connect: Connection refused (error 111)
I20260928 08:40:23.616050 13981 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:37747
I20260928 08:40:23.616089 14047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:37747 every 8 connection(s)
I20260928 08:40:23.617362 13981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-2/data/info.pb
I20260928 08:40:23.621003 14048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:40:23.623715 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 13981
I20260928 08:40:23.623924 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584618551208-388-0/minicluster-data/master-2/wal/instance
I20260928 08:40:23.624434 388 external_mini_cluster.cc:1428] Setting key fb172d721c7ddf42d9962d256e9c55b4
I20260928 08:40:23.628913 14048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:40:23 dist-test-slave-w6c0 krb5kdc[13820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 28 08:40:23 dist-test-slave-w6c0 krb5kdc[13820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:23.647001 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.631353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41206 (local address 127.0.97.62:41493)
0928 08:40:23.631579 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:23.631585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:23.632154 (+ 569us) server_negotiation.cc:407] Connection header received
0928 08:40:23.632326 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:23.632331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:23.632403 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:23.632501 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:23.633492 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.634168 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.635711 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.636458 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.636607 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.639710 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:23.639716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:23.639721 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:23.639760 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:23.643050 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.643954 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.643959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.643963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.644145 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.644626 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.644632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.644635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.644976 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:23.645244 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:23.646380 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0928 08:40:23.646645 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:40:23.647481 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.626303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58450 (local address 127.0.97.60:37747)
0928 08:40:23.627080 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:23.627092 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:23.627130 (+ 38us) server_negotiation.cc:407] Connection header received
0928 08:40:23.628699 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:23.628719 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:23.629072 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:23.629471 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:23.630927 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.632615 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.634002 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.634733 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.634950 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.639496 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:23.639504 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:23.639515 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:23.639546 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:23.643117 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.643954 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.643962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.643965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.644168 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.644561 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.644585 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.644591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.645232 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:23.645521 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:23.645852 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 08:40:23.646162 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":581,"thread_start_us":354,"threads_started":1}
I20260928 08:40:23.647514 14052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.631291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41493 (local address 127.0.0.1:41206)
0928 08:40:23.631998 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.632051 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:23.632206 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:23.632550 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:23.632555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:23.633081 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:23.633328 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:23.633336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.634339 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.634344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:23.635574 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:23.635591 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.636653 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.636658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:23.636772 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.637382 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:23.637403 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:23.639515 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:23.643246 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:23.643258 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:23.643269 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:23.643818 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:23.644279 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:23.644284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:23.644287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:23.644514 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:23.645224 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:23.645232 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:23.645470 (+ 238us) client_negotiation.cc:769] Sending connection context
0928 08:40:23.645732 (+ 262us) client_negotiation.cc:240] Negotiation successful
0928 08:40:23.646729 (+ 997us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":536,"thread_start_us":341,"threads_started":1}
I20260928 08:40:23.649326 14048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:40:23 dist-test-slave-w6c0 krb5kdc[13820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 28 08:40:23 dist-test-slave-w6c0 krb5kdc[13820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:23.663615 14052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.650144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36877 (local address 127.0.0.1:49476)
0928 08:40:23.650286 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.650306 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:23.650399 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:23.651084 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:23.651089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:23.651526 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:23.651706 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:23.651718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.653104 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.653109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:23.653970 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:23.653980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.654907 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.654913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:23.655134 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.655748 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:23.655765 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:23.657661 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:23.661234 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:23.661238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:23.661240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:23.661583 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:23.661992 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:23.661995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:23.661997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:23.662135 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:23.662695 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:23.662699 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:23.662837 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 08:40:23.662995 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 08:40:23.663260 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 08:40:23.663661 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.650555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.0.97.61:36877)
0928 08:40:23.650747 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:23.650754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:23.650773 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:40:23.650833 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:23.650838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:23.650924 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:23.651027 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:23.651852 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.652973 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.654116 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.654742 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.654868 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.657853 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:23.657861 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:23.657868 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:23.657913 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:23.661108 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.661702 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.661709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.661713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.661858 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.662242 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.662248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.662252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.662577 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:23.662715 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:23.663084 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 08:40:23.663289 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:40:23.664767 14048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:40:23.666790 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.651260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41208 (local address 127.0.97.62:41493)
0928 08:40:23.651421 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:23.651426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:23.651441 (+ 15us) server_negotiation.cc:407] Connection header received
0928 08:40:23.651477 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:23.651480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:23.651527 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:23.651631 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:23.653050 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.654208 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.655765 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.656203 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.656334 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.660920 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:23.660927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:23.660931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:23.660970 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:23.663633 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.664618 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.664624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.664629 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.664777 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.665188 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.665191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.665193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.665584 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:23.665900 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:23.666184 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 08:40:23.666417 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 08:40:23 dist-test-slave-w6c0 krb5kdc[13820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 28 08:40:23 dist-test-slave-w6c0 krb5kdc[13820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 08:40:23.677686 14052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.665669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58458)
0928 08:40:23.665857 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.665875 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:23.665970 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:23.666319 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:23.666325 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:23.666747 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:23.666918 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:23.666933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.667995 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.667999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:23.669238 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:23.669248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.669922 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.669926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:23.670103 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.670644 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:23.670660 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:23.672472 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:23.675325 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:23.675333 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:23.675336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:23.675672 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:23.676056 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:23.676061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:23.676067 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:23.676193 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:23.676661 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:23.676666 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:23.676802 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 08:40:23.676946 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 08:40:23.677264 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 08:40:23.677731 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.665899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58458 (local address 127.0.97.60:37747)
0928 08:40:23.666027 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:23.666030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:23.666043 (+ 13us) server_negotiation.cc:407] Connection header received
0928 08:40:23.666106 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:23.666110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:23.666173 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:23.666271 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:23.667051 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.667866 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.669368 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.669791 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.669917 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.672664 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:23.672669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:23.672672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:23.672697 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:23.675204 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.675782 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.675790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.675793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.675954 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.676292 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.676300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.676304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.676552 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:23.676698 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:23.677035 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 08:40:23.677406 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":162,"server-negotiator.queue_time_us":47}
I20260928 08:40:23.682184 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.668309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49484 (local address 127.0.97.61:36877)
0928 08:40:23.668445 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:23.668456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:23.668471 (+ 15us) server_negotiation.cc:407] Connection header received
0928 08:40:23.668616 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:23.668620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:23.668689 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:23.668782 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:23.669912 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.670783 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.671946 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.672421 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.672561 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.676679 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:23.676684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:23.676687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:23.676716 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:23.679621 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.680255 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.680263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.680268 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.680415 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.680820 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.680825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.680829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.681091 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:23.681358 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:23.681624 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 08:40:23.681873 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 08:40:23.683183 14048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a: Bootstrap starting.
I20260928 08:40:23.687021 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:40:23.687067 14048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a: Neither blocks nor log segments found. Creating new log.
I20260928 08:40:23.688074 14048 log.cc:824] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:23.691099 14048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a: No bootstrap required, opened a new log
I20260928 08:40:23.703223 14048 raft_consensus.cc:348] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } peers { permanent_uuid: "3e244e01425647c4bb93fc1286232581" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } } peers { permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } }
I20260928 08:40:23.703716 14048 raft_consensus.cc:374] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:40:23.703873 14048 raft_consensus.cc:729] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a63dad4209a54b08953a8dcf389d428a, State: Initialized, Role: FOLLOWER
I20260928 08:40:23.705679 14048 consensus_queue.cc:260] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } peers { permanent_uuid: "3e244e01425647c4bb93fc1286232581" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } } peers { permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } }
I20260928 08:40:23.709527 14048 sys_catalog.cc:564] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:40:23.713752 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.691943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58464 (local address 127.0.97.60:37747)
0928 08:40:23.692106 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:23.692114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:23.694129 (+ 2015us) server_negotiation.cc:407] Connection header received
0928 08:40:23.694354 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:23.694360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:23.694426 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:23.694529 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:23.696598 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.697809 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.700141 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.700871 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.700985 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.705289 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:23.705296 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:23.705300 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:23.705337 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:23.709209 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.710173 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.710179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.710182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.710340 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.710818 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.710826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.710841 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.711381 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:23.711693 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:23.712918 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0928 08:40:23.713242 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:40:23.714248 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } peers { permanent_uuid: "3e244e01425647c4bb93fc1286232581" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } } peers { permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } } }
I20260928 08:40:23.714684 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:40:23.714802 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.692386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41212 (local address 127.0.97.62:41493)
0928 08:40:23.692577 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:23.692584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:23.693456 (+ 872us) server_negotiation.cc:407] Connection header received
0928 08:40:23.693934 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:23.693943 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:23.694045 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:23.694134 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:23.697238 (+ 3104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.698677 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.701323 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.702226 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.702348 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.707268 (+ 4920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:23.707279 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:23.707286 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:23.707343 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:23.711413 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.712306 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.712311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.712315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.712482 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.713054 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.713060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.713064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.713412 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:23.713638 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:23.714167 (+ 529us) server_negotiation.cc:299] Negotiation successful
0928 08:40:23.714405 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:40:23.726003 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.692198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.0.97.61:36877)
0928 08:40:23.692390 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:23.692395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:23.708218 (+ 15823us) server_negotiation.cc:407] Connection header received
0928 08:40:23.708772 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:23.708779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:23.708876 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:23.708959 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:23.711278 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.712411 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.714651 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.715267 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.715374 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.718583 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:23.718592 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:23.718600 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:23.718649 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:23.722440 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.723093 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.723098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.723102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.723290 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.723740 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.723746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.723749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.724139 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:23.725256 (+ 1117us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:23.725392 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 08:40:23.725641 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 08:40:23.727252 14072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:40:23.727377 14072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:40:23.724092 13981 master_runner.cc:186] Error getting master registration for 127.0.97.62:41493: OK, instance_id { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" instance_seqno: 1790584822640368 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:40:23.732393 13981 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 08:40:23.781895 13900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } attempt: 4
Sep 28 08:40:23 dist-test-slave-w6c0 krb5kdc[13820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 08:40:23.797003 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.782634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58472 (local address 127.0.97.60:37747)
0928 08:40:23.782824 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:23.782831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:23.782851 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:40:23.783047 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:23.783054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:23.783149 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:23.783231 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:23.784212 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.784973 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.786746 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.787165 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.787260 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.791607 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:23.791612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:23.791615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:23.791643 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:23.794404 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.795188 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.795192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.795195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.795331 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.795743 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.795746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.795748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.796010 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:23.796120 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:23.796546 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 08:40:23.796735 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 08:40:23.797178 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.782553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58472)
0928 08:40:23.782768 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.782792 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:23.782903 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:23.783291 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:23.783296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:23.783835 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:23.784057 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:23.784070 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.785098 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.785104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:23.786620 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:23.786635 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.787307 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.787314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:23.787507 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.788187 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:23.788211 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:23.791431 (+ 3220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:23.794564 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:23.794572 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:23.794577 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:23.795073 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:23.795442 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:23.795447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:23.795451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:23.795638 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:23.796123 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:23.796129 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:23.796300 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 08:40:23.796444 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 08:40:23.796781 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 08:40:23.802487 13900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878: Bootstrap starting.
I20260928 08:40:23.806589 13900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878: Neither blocks nor log segments found. Creating new log.
I20260928 08:40:23.807865 13900 log.cc:824] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:23.810714 13900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878: No bootstrap required, opened a new log
I20260928 08:40:23.818343 13900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } peers { permanent_uuid: "3e244e01425647c4bb93fc1286232581" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } } peers { permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } }
I20260928 08:40:23.818724 13900 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:40:23.818859 13900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b2eea2a968641c18dc3b80539f9f878, State: Initialized, Role: FOLLOWER
I20260928 08:40:23.819463 13900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } peers { permanent_uuid: "3e244e01425647c4bb93fc1286232581" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } } peers { permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } }
I20260928 08:40:23.820402 14075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } peers { permanent_uuid: "3e244e01425647c4bb93fc1286232581" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } } peers { permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } } }
I20260928 08:40:23.820779 14075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:40:23.821617 13900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 08:40:23.831292 14086 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:40:23.831416 14086 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:40:23.830710 13833 master_runner.cc:186] Error getting master registration for 127.0.97.61:36877: OK, instance_id { permanent_uuid: "3e244e01425647c4bb93fc1286232581" instance_seqno: 1790584823120190 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:40:23.843204 13833 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 08:40:23.847527 13974 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } attempt: 5
W20260928 08:40:23.849634 14002 tablet.cc:2369] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 08:40:23 dist-test-slave-w6c0 krb5kdc[13820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 08:40:23.865916 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.848208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37747 (local address 127.0.0.1:58480)
0928 08:40:23.848495 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:23.848516 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:23.848659 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:23.849046 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:23.849051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:23.849586 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:23.849797 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:23.849809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.851462 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.851468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:23.852944 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:23.852960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.855051 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.855057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:23.855293 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.856042 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:23.856070 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:23.858889 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:23.863122 (+ 4233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:23.863128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:23.863132 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:23.863660 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:23.864068 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:23.864073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:23.864077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:23.864236 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:23.864898 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:23.864903 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:23.865065 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 08:40:23.865221 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 08:40:23.865505 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260928 08:40:23.865993 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:23.848314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58480 (local address 127.0.97.60:37747)
0928 08:40:23.848497 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:23.848503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:23.848611 (+ 108us) server_negotiation.cc:407] Connection header received
0928 08:40:23.848795 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:23.848801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:23.848883 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:23.849001 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:23.850032 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.851302 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.854237 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:23.854892 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:23.855020 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:23.859143 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:23.859151 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:23.859156 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:23.859210 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:23.862975 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.863787 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.863793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.863797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.863941 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:23.864356 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:23.864362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:23.864366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:23.864758 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:23.864924 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:23.865337 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 08:40:23.865615 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:40:23.872406 13974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581: Bootstrap starting.
I20260928 08:40:23.877121 13974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581: Neither blocks nor log segments found. Creating new log.
I20260928 08:40:23.878425 13974 log.cc:824] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581: Log is configured to *not* fsync() on all Append() calls
W20260928 08:40:23.879977 13854 tablet.cc:2369] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:40:23.882594 13974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581: No bootstrap required, opened a new log
I20260928 08:40:23.893602 13974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } peers { permanent_uuid: "3e244e01425647c4bb93fc1286232581" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } } peers { permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } }
I20260928 08:40:23.893993 13974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:40:23.894091 13974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e244e01425647c4bb93fc1286232581, State: Initialized, Role: FOLLOWER
I20260928 08:40:23.894752 13974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } peers { permanent_uuid: "3e244e01425647c4bb93fc1286232581" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } } peers { permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } }
I20260928 08:40:23.896713 14089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } peers { permanent_uuid: "3e244e01425647c4bb93fc1286232581" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } } peers { permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } } }
I20260928 08:40:23.897204 14089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:40:23.898083 13974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:40:23.912441 13907 master_runner.cc:263] No leader master found from master 3e244e01425647c4bb93fc1286232581
I20260928 08:40:23.912600 13907 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 08:40:23.913679 14100 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:40:23.913869 14100 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:40:23.947543 14089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:40:23.947908 14089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } peers { permanent_uuid: "3e244e01425647c4bb93fc1286232581" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } } peers { permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } }
I20260928 08:40:23.949555 14089 leader_election.cc:288] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b2eea2a968641c18dc3b80539f9f878 (127.0.97.62:41493), a63dad4209a54b08953a8dcf389d428a (127.0.97.60:37747)
I20260928 08:40:23.951143 13875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e244e01425647c4bb93fc1286232581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b2eea2a968641c18dc3b80539f9f878" is_pre_election: true
I20260928 08:40:23.951761 13875 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e244e01425647c4bb93fc1286232581 in term 0.
I20260928 08:40:23.952399 14023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e244e01425647c4bb93fc1286232581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a63dad4209a54b08953a8dcf389d428a" is_pre_election: true
I20260928 08:40:23.952936 14023 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e244e01425647c4bb93fc1286232581 in term 0.
I20260928 08:40:23.953749 13923 leader_election.cc:302] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e244e01425647c4bb93fc1286232581, 4b2eea2a968641c18dc3b80539f9f878; no voters:
I20260928 08:40:23.954710 14089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:40:23.954859 14089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:40:23.954952 14089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:40:23.958370 14089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } peers { permanent_uuid: "3e244e01425647c4bb93fc1286232581" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } } peers { permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } }
I20260928 08:40:23.959614 13875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e244e01425647c4bb93fc1286232581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b2eea2a968641c18dc3b80539f9f878"
I20260928 08:40:23.959838 13875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:40:23.972106 13875 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e244e01425647c4bb93fc1286232581 in term 1.
I20260928 08:40:23.974517 13923 leader_election.cc:302] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e244e01425647c4bb93fc1286232581, 4b2eea2a968641c18dc3b80539f9f878; no voters:
I20260928 08:40:23.979125 14023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e244e01425647c4bb93fc1286232581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a63dad4209a54b08953a8dcf389d428a"
I20260928 08:40:23.979367 14023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:40:23.985975 14023 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e244e01425647c4bb93fc1286232581 in term 1.
I20260928 08:40:23.986868 14089 leader_election.cc:288] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [CANDIDATE]: Term 1 election: Requested vote from peers 4b2eea2a968641c18dc3b80539f9f878 (127.0.97.62:41493), a63dad4209a54b08953a8dcf389d428a (127.0.97.60:37747)
I20260928 08:40:23.987324 14089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:40:23.987855 14089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [term 1 LEADER]: Becoming Leader. State: Replica: 3e244e01425647c4bb93fc1286232581, State: Running, Role: LEADER
I20260928 08:40:23.988998 14089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } peers { permanent_uuid: "3e244e01425647c4bb93fc1286232581" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } } peers { permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } }
I20260928 08:40:24.009486 14102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e244e01425647c4bb93fc1286232581. Latest consensus state: current_term: 1 leader_uuid: "3e244e01425647c4bb93fc1286232581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } peers { permanent_uuid: "3e244e01425647c4bb93fc1286232581" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } } peers { permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } } }
I20260928 08:40:24.010161 14102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [sys.catalog]: This master's current role is: LEADER
I20260928 08:40:24.015166 14104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:40:24.017542 14104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:40:24.060664 13875 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878 [term 1 FOLLOWER]: Refusing update from remote peer 3e244e01425647c4bb93fc1286232581: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:40:24.061806 14023 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a [term 1 FOLLOWER]: Refusing update from remote peer 3e244e01425647c4bb93fc1286232581: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:40:24.062996 14102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:40:24.064450 14102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:40:24.090090 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e244e01425647c4bb93fc1286232581. Latest consensus state: current_term: 1 leader_uuid: "3e244e01425647c4bb93fc1286232581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } peers { permanent_uuid: "3e244e01425647c4bb93fc1286232581" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } } peers { permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } } }
I20260928 08:40:24.090431 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:40:24.094262 14105 mvcc.cc:205] Tried to move back new op lower bound from 7334235439287791616 to 7334235439069859840. Current Snapshot: MvccSnapshot[applied={T|T < 7334235439287791616}]
I20260928 08:40:24.099279 14102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e244e01425647c4bb93fc1286232581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } peers { permanent_uuid: "3e244e01425647c4bb93fc1286232581" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } } peers { permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } } }
I20260928 08:40:24.099769 14102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [sys.catalog]: This master's current role is: LEADER
I20260928 08:40:24.104530 14075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e244e01425647c4bb93fc1286232581. Latest consensus state: current_term: 1 leader_uuid: "3e244e01425647c4bb93fc1286232581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } peers { permanent_uuid: "3e244e01425647c4bb93fc1286232581" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } } peers { permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } } }
I20260928 08:40:24.104882 14075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:40:24.106839 14104 catalog_manager.cc:1347] Generated new cluster ID: 272b8ad60ffb4d0894c60704b664c4c0
I20260928 08:40:24.106993 14104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:40:24.108744 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e244e01425647c4bb93fc1286232581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } peers { permanent_uuid: "3e244e01425647c4bb93fc1286232581" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } } peers { permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } } }
I20260928 08:40:24.109000 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:40:24.126194 14102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e244e01425647c4bb93fc1286232581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } peers { permanent_uuid: "3e244e01425647c4bb93fc1286232581" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } } peers { permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } } }
I20260928 08:40:24.126608 14102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581 [sys.catalog]: This master's current role is: LEADER
I20260928 08:40:24.136247 14075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e244e01425647c4bb93fc1286232581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2eea2a968641c18dc3b80539f9f878" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } peers { permanent_uuid: "3e244e01425647c4bb93fc1286232581" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36877 } } peers { permanent_uuid: "a63dad4209a54b08953a8dcf389d428a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37747 } } }
I20260928 08:40:24.136512 14075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:40:24.152084 14104 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:40:24.154492 14104 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:40:24.189733 14104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e244e01425647c4bb93fc1286232581: Generated new TSK 0
I20260928 08:40:24.190953 14104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:40:24.729328 14072 catalog_manager.cc:1259] Loaded cluster ID: 272b8ad60ffb4d0894c60704b664c4c0
I20260928 08:40:24.729435 14072 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a: loading cluster ID for follower catalog manager: success
I20260928 08:40:24.731933 14072 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a: acquiring CA information for follower catalog manager: success
I20260928 08:40:24.733709 14072 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a63dad4209a54b08953a8dcf389d428a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:40:24.834631 14086 catalog_manager.cc:1259] Loaded cluster ID: 272b8ad60ffb4d0894c60704b664c4c0
I20260928 08:40:24.834735 14086 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878: loading cluster ID for follower catalog manager: success
I20260928 08:40:24.837270 14086 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878: acquiring CA information for follower catalog manager: success
I20260928 08:40:24.839111 14086 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4b2eea2a968641c18dc3b80539f9f878: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:40:25.337496 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 13833
I20260928 08:40:25.338795 13895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:25.451573 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 13833
I20260928 08:40:25.472625 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 13907
I20260928 08:40:25.473681 13969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:25.582404 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 13907
W20260928 08:40:25.602711 14000 connection.cc:535] server connection from 127.0.0.1:58480 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 08:40:25.603338 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 13981
I20260928 08:40:25.604298 14043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:25.711553 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 13981
2026-09-28T08:40:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4815 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:40:25 dist-test-slave-w6c0 krb5kdc[14124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:40:25 dist-test-slave-w6c0 krb5kdc[14124](info): set up 2 sockets
Sep 28 08:40:25 dist-test-slave-w6c0 krb5kdc[14124](info): commencing operation
krb5kdc: starting...
W20260928 08:40:26.802951 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:40:26 dist-test-slave-w6c0 krb5kdc[14124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:40:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:40:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:40:27.042970 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35521
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:40:27.374203 14137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:27.374476 14137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:27.374662 14137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:27.382085 14137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:40:27.382202 14137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:27.382324 14137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:27.382432 14137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:40:27.382534 14137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:40:27.392629 14137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:27.393965 14137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:27.395388 14137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:27.407491 14145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:27.407483 14143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:27.407528 14142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:27.408345 14137 server_base.cc:1034] running on GCE node
I20260928 08:40:27.409233 14137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:27.410772 14137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:27.412011 14137 hybrid_clock.cc:648] HybridClock initialized: now 1790584827411975 us; error 46 us; skew 500 ppm
I20260928 08:40:27.412375 14137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:27 dist-test-slave-w6c0 krb5kdc[14124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:27.415647 14137 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:40:27.417097 14137 webserver.cc:460] Webserver started at http://127.0.97.62:39725/ using document root <none> and password file <none>
I20260928 08:40:27.417788 14137 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:27.417888 14137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:27.418160 14137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:40:27.421185 14137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "62110a568b484ed79ed19701541ddc99"
format_stamp: "Formatted at 2026-09-28 08:40:27 on dist-test-slave-w6c0"
server_key: "1b5b051ef7eeffe72b6535e0ca92593a"
server_key_iv: "9ed6981352cc9a035e86b0828a2aa6e8"
server_key_version: "encryptionkey@0"
I20260928 08:40:27.421890 14137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "62110a568b484ed79ed19701541ddc99"
format_stamp: "Formatted at 2026-09-28 08:40:27 on dist-test-slave-w6c0"
server_key: "1b5b051ef7eeffe72b6535e0ca92593a"
server_key_iv: "9ed6981352cc9a035e86b0828a2aa6e8"
server_key_version: "encryptionkey@0"
I20260928 08:40:27.427161 14137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 08:40:27.430706 14152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:27.432121 14137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:40:27.432281 14137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "62110a568b484ed79ed19701541ddc99"
format_stamp: "Formatted at 2026-09-28 08:40:27 on dist-test-slave-w6c0"
server_key: "1b5b051ef7eeffe72b6535e0ca92593a"
server_key_iv: "9ed6981352cc9a035e86b0828a2aa6e8"
server_key_version: "encryptionkey@0"
I20260928 08:40:27.432454 14137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:27.454008 14137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:27.458577 14137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:27.458897 14137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:27.476338 14137 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35521
I20260928 08:40:27.476382 14203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35521 every 8 connection(s)
I20260928 08:40:27.477763 14137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:40:27.481498 14204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:40:27.486147 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 14137
I20260928 08:40:27.486351 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:40:27.487238 388 external_mini_cluster.cc:1428] Setting key 31712f34ddc4d5cd014f1fcae0b87310
I20260928 08:40:27.494375 14204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99: Bootstrap starting.
I20260928 08:40:27.498044 14204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99: Neither blocks nor log segments found. Creating new log.
I20260928 08:40:27.499292 14204 log.cc:824] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:27.502177 14204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99: No bootstrap required, opened a new log
Sep 28 08:40:27 dist-test-slave-w6c0 krb5kdc[14124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:27.513087 14207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:27.489095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.0.97.62:35521)
0928 08:40:27.489876 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:27.489892 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:27.489941 (+ 49us) server_negotiation.cc:407] Connection header received
0928 08:40:27.492699 (+ 2758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:27.492734 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:27.493203 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:27.493618 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:27.495043 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:27.497099 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:27.498537 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:27.499568 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:27.499778 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:27.503994 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:27.504003 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:27.504022 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:27.504074 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:27.508326 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:27.509069 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:27.509077 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:27.509086 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:27.509300 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:27.509661 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:27.509679 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:27.509683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:27.510428 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:27.510788 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:27.511168 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 08:40:27.511690 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":311,"threads_started":1}
I20260928 08:40:27.513046 14204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62110a568b484ed79ed19701541ddc99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35521 } }
I20260928 08:40:27.513537 14204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:40:27.513659 14204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62110a568b484ed79ed19701541ddc99, State: Initialized, Role: FOLLOWER
I20260928 08:40:27.514254 14204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62110a568b484ed79ed19701541ddc99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35521 } }
I20260928 08:40:27.514546 14204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:40:27.514710 14204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:40:27.514911 14204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:40:27.517230 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:40:27.517701 14204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62110a568b484ed79ed19701541ddc99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35521 } }
I20260928 08:40:27.518206 14204 leader_election.cc:302] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62110a568b484ed79ed19701541ddc99; no voters:
I20260928 08:40:27.518936 14204 leader_election.cc:288] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:40:27.519029 14209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:40:27.520726 14209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99 [term 1 LEADER]: Becoming Leader. State: Replica: 62110a568b484ed79ed19701541ddc99, State: Running, Role: LEADER
I20260928 08:40:27.521283 14204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:40:27.521311 14209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62110a568b484ed79ed19701541ddc99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35521 } }
I20260928 08:40:27.524315 14211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62110a568b484ed79ed19701541ddc99. Latest consensus state: current_term: 1 leader_uuid: "62110a568b484ed79ed19701541ddc99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62110a568b484ed79ed19701541ddc99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35521 } } }
I20260928 08:40:27.524698 14211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99 [sys.catalog]: This master's current role is: LEADER
I20260928 08:40:27.524399 14210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62110a568b484ed79ed19701541ddc99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62110a568b484ed79ed19701541ddc99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35521 } } }
I20260928 08:40:27.524807 14210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99 [sys.catalog]: This master's current role is: LEADER
I20260928 08:40:27.526546 14216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:40:27.532202 14216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:40:27.540678 14216 catalog_manager.cc:1347] Generated new cluster ID: 2d18beecdff747c891d63004ba8746ca
I20260928 08:40:27.540802 14216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:40:27.555454 14216 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:40:27.556769 14216 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:40:27.571151 14216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62110a568b484ed79ed19701541ddc99: Generated new TSK 0
I20260928 08:40:27.572216 14216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:40:27.629516 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 14137
I20260928 08:40:27.630774 14199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:27.730145 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 14137
2026-09-28T08:40:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2014 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:40:27 dist-test-slave-w6c0 krb5kdc[14232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:40:27 dist-test-slave-w6c0 krb5kdc[14232](info): set up 2 sockets
Sep 28 08:40:27 dist-test-slave-w6c0 krb5kdc[14232](info): commencing operation
krb5kdc: starting...
W20260928 08:40:28.834949 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:40:28 dist-test-slave-w6c0 krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:40:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:40:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:40:29.068157 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37999
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:40:29.398447 14245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:29.398916 14245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:29.399078 14245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:29.406616 14245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:40:29.406792 14245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:29.406924 14245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:29.407022 14245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:40:29.407115 14245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:40:29.417415 14245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:29.419348 14245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:29.420691 14245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:29.433159 14250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:29.433182 14251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:29.433334 14253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:29.433943 14245 server_base.cc:1034] running on GCE node
I20260928 08:40:29.434746 14245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:29.436146 14245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:29.437377 14245 hybrid_clock.cc:648] HybridClock initialized: now 1790584829437332 us; error 67 us; skew 500 ppm
I20260928 08:40:29.437731 14245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:29 dist-test-slave-w6c0 krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:29.441013 14245 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:40:29.442476 14245 webserver.cc:460] Webserver started at http://127.0.97.62:43905/ using document root <none> and password file <none>
I20260928 08:40:29.443221 14245 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:29.443367 14245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:29.443670 14245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:40:29.446707 14245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "b8fa88119c1246d38924104841d045eb"
format_stamp: "Formatted at 2026-09-28 08:40:29 on dist-test-slave-w6c0"
server_key: "62a0b985cf4bfd74ab700a60117241e2"
server_key_iv: "d4dfd2a6cf931fada22ec07352b7338f"
server_key_version: "encryptionkey@0"
I20260928 08:40:29.447474 14245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "b8fa88119c1246d38924104841d045eb"
format_stamp: "Formatted at 2026-09-28 08:40:29 on dist-test-slave-w6c0"
server_key: "62a0b985cf4bfd74ab700a60117241e2"
server_key_iv: "d4dfd2a6cf931fada22ec07352b7338f"
server_key_version: "encryptionkey@0"
I20260928 08:40:29.452768 14245 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:40:29.456215 14260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:29.457664 14245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:40:29.457847 14245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "b8fa88119c1246d38924104841d045eb"
format_stamp: "Formatted at 2026-09-28 08:40:29 on dist-test-slave-w6c0"
server_key: "62a0b985cf4bfd74ab700a60117241e2"
server_key_iv: "d4dfd2a6cf931fada22ec07352b7338f"
server_key_version: "encryptionkey@0"
I20260928 08:40:29.458055 14245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:29.473981 14245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:29.480213 14245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:29.480616 14245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:29.500749 14245 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37999
I20260928 08:40:29.500772 14311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37999 every 8 connection(s)
I20260928 08:40:29.502208 14245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:40:29.506156 14312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:40:29.510960 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 14245
I20260928 08:40:29.511183 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:40:29.511829 388 external_mini_cluster.cc:1428] Setting key 488a93afe561d75e815a204a3b586bc8
I20260928 08:40:29.518602 14312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb: Bootstrap starting.
I20260928 08:40:29.522439 14312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb: Neither blocks nor log segments found. Creating new log.
I20260928 08:40:29.523499 14312 log.cc:824] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:29.526425 14312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb: No bootstrap required, opened a new log
Sep 28 08:40:29 dist-test-slave-w6c0 krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:29.534605 14312 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fa88119c1246d38924104841d045eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37999 } }
I20260928 08:40:29.535009 14312 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:40:29.535149 14312 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8fa88119c1246d38924104841d045eb, State: Initialized, Role: FOLLOWER
I20260928 08:40:29.535928 14312 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fa88119c1246d38924104841d045eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37999 } }
I20260928 08:40:29.536193 14312 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:40:29.536355 14312 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:40:29.536541 14312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:40:29.536831 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:29.513982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.0.97.62:37999)
0928 08:40:29.515039 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:29.515053 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:29.515104 (+ 51us) server_negotiation.cc:407] Connection header received
0928 08:40:29.517503 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:29.517534 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:29.518077 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:29.518572 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:29.520028 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:29.522088 (+ 2060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:29.523812 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:29.524690 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:29.524905 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:29.529210 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:29.529218 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:29.529234 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:29.529274 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:29.532526 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:29.533248 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:29.533260 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:29.533268 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:29.533430 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:29.533849 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:29.533853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:29.533855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:29.534412 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:29.534736 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:29.535170 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 08:40:29.535587 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":521,"threads_started":1}
I20260928 08:40:29.539288 14312 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fa88119c1246d38924104841d045eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37999 } }
I20260928 08:40:29.539856 14312 leader_election.cc:302] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8fa88119c1246d38924104841d045eb; no voters:
I20260928 08:40:29.540424 14312 leader_election.cc:288] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:40:29.540576 14317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:40:29.541262 14317 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 1 LEADER]: Becoming Leader. State: Replica: b8fa88119c1246d38924104841d045eb, State: Running, Role: LEADER
I20260928 08:40:29.541793 14317 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fa88119c1246d38924104841d045eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37999 } }
I20260928 08:40:29.542598 14312 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:40:29.544523 14318 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8fa88119c1246d38924104841d045eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fa88119c1246d38924104841d045eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37999 } } }
I20260928 08:40:29.544999 14318 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [sys.catalog]: This master's current role is: LEADER
I20260928 08:40:29.545513 14319 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8fa88119c1246d38924104841d045eb. Latest consensus state: current_term: 1 leader_uuid: "b8fa88119c1246d38924104841d045eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fa88119c1246d38924104841d045eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37999 } } }
I20260928 08:40:29.545816 14319 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [sys.catalog]: This master's current role is: LEADER
I20260928 08:40:29.547473 14322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:40:29.553153 14322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:40:29.562287 14322 catalog_manager.cc:1347] Generated new cluster ID: 52e5b7e5c4474079a1ea06cd935f0126
I20260928 08:40:29.562490 14322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:40:29.584518 14322 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:40:29.586539 14322 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:40:29.600014 14322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb: Generated new TSK 0
I20260928 08:40:29.601133 14322 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:40:29.658344 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37999
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:40:29.987828 14338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:29.988112 14338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:29.988265 14338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:29.995566 14338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:29.995697 14338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:29.995874 14338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:40:30.005596 14338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:30.006953 14338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:30.008255 14338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:30.021970 14343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:30.021992 14346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:30.021907 14344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:30.022843 14338 server_base.cc:1034] running on GCE node
I20260928 08:40:30.023458 14338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:30.024394 14338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:30.025626 14338 hybrid_clock.cc:648] HybridClock initialized: now 1790584830025588 us; error 50 us; skew 500 ppm
I20260928 08:40:30.025959 14338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:30 dist-test-slave-w6c0 krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:30.029168 14338 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:40:30.030786 14338 webserver.cc:460] Webserver started at http://127.0.97.1:45327/ using document root <none> and password file <none>
I20260928 08:40:30.031468 14338 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:30.031595 14338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:30.031900 14338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:40:30.034838 14338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "30b41b45ec8a4756acef459e9b67168e"
format_stamp: "Formatted at 2026-09-28 08:40:30 on dist-test-slave-w6c0"
server_key: "124cd31eb95c3b9d135eff94a7e6bbfb"
server_key_iv: "bd06d33696e1bf12cd6c19d0146357af"
server_key_version: "encryptionkey@0"
I20260928 08:40:30.035614 14338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "30b41b45ec8a4756acef459e9b67168e"
format_stamp: "Formatted at 2026-09-28 08:40:30 on dist-test-slave-w6c0"
server_key: "124cd31eb95c3b9d135eff94a7e6bbfb"
server_key_iv: "bd06d33696e1bf12cd6c19d0146357af"
server_key_version: "encryptionkey@0"
I20260928 08:40:30.040846 14338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:40:30.044498 14353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:30.045728 14338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:40:30.045895 14338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "30b41b45ec8a4756acef459e9b67168e"
format_stamp: "Formatted at 2026-09-28 08:40:30 on dist-test-slave-w6c0"
server_key: "124cd31eb95c3b9d135eff94a7e6bbfb"
server_key_iv: "bd06d33696e1bf12cd6c19d0146357af"
server_key_version: "encryptionkey@0"
I20260928 08:40:30.046130 14338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:30.068364 14338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:30.072574 14338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:30.072822 14338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:30.073742 14338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:40:30.075062 14338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:40:30.075138 14338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:30.075222 14338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:40:30.075266 14338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:30.107967 14338 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33225
I20260928 08:40:30.108016 14465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33225 every 8 connection(s)
I20260928 08:40:30.109217 14338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:40:30.111826 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 14338
I20260928 08:40:30.112047 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:40:30.112775 388 external_mini_cluster.cc:1428] Setting key 3866f934937611b73974d5be8dcc91d1
Sep 28 08:40:30 dist-test-slave-w6c0 krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:30.129544 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:30.111818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55971 (local address 127.0.97.62:37999)
0928 08:40:30.112358 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:30.112366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:30.113296 (+ 930us) server_negotiation.cc:407] Connection header received
0928 08:40:30.115420 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:30.115425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:30.115515 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:30.115630 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:30.117082 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:30.117805 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:30.119452 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:30.119856 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:30.119991 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:30.123975 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:30.123979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:30.123983 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:30.124014 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:30.126472 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:30.127227 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:30.127232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:30.127238 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:30.127361 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:30.127781 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:30.127786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:30.127790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:30.128064 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:30.128233 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:30.128927 (+ 694us) server_negotiation.cc:299] Negotiation successful
0928 08:40:30.129150 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":240,"threads_started":1}
I20260928 08:40:30.130465 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:30.112329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:55971)
0928 08:40:30.113122 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:30.113172 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:30.115156 (+ 1984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:30.115868 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:30.115876 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:30.116361 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:30.116912 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:30.116932 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:30.117988 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:30.117992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:30.119293 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:30.119306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:30.120096 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:30.120103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:30.120327 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:30.121103 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:30.121138 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:30.123747 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:30.126659 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:30.126667 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:30.126683 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:30.127100 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:30.127472 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:30.127478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:30.127481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:30.127661 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:30.128191 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:30.128200 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:30.128551 (+ 351us) client_negotiation.cc:769] Sending connection context
0928 08:40:30.128808 (+ 257us) client_negotiation.cc:240] Negotiation successful
0928 08:40:30.129316 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":329,"threads_started":1}
I20260928 08:40:30.132261 14466 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37999
I20260928 08:40:30.132594 14466 heartbeater.cc:464] Registering TS with master...
I20260928 08:40:30.133293 14466 heartbeater.cc:511] Master 127.0.97.62:37999 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:40:30.135515 14277 ts_manager.cc:194] Registered new tserver with Master: 30b41b45ec8a4756acef459e9b67168e (127.0.97.1:33225)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 08:40:30.137287 14277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55971
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:40:30.166450 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37999
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:40:30.490634 14472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:30.490914 14472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:30.491009 14472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:30.498332 14472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:30.498437 14472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:30.498626 14472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:40:30.513427 14472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:30.515162 14472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:30.516762 14472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:30.531400 14477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:30.531347 14478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:30.531538 14480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:30.532512 14472 server_base.cc:1034] running on GCE node
I20260928 08:40:30.533150 14472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:30.534165 14472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:30.535394 14472 hybrid_clock.cc:648] HybridClock initialized: now 1790584830535358 us; error 59 us; skew 500 ppm
I20260928 08:40:30.535741 14472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:30 dist-test-slave-w6c0 krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:30.539093 14472 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:40:30.540601 14472 webserver.cc:460] Webserver started at http://127.0.97.2:38915/ using document root <none> and password file <none>
I20260928 08:40:30.541337 14472 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:30.541491 14472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:30.541810 14472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:40:30.545897 14472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "5e1374b04d764dc291f07654df6c6291"
format_stamp: "Formatted at 2026-09-28 08:40:30 on dist-test-slave-w6c0"
server_key: "bd3ce6e53194857d497e8aa7c67a07a3"
server_key_iv: "0be4080e633e804677b17c317fc065f1"
server_key_version: "encryptionkey@0"
I20260928 08:40:30.547212 14472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5e1374b04d764dc291f07654df6c6291"
format_stamp: "Formatted at 2026-09-28 08:40:30 on dist-test-slave-w6c0"
server_key: "bd3ce6e53194857d497e8aa7c67a07a3"
server_key_iv: "0be4080e633e804677b17c317fc065f1"
server_key_version: "encryptionkey@0"
I20260928 08:40:30.553135 14472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:40:30.556838 14487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:30.558094 14472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:40:30.558288 14472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "5e1374b04d764dc291f07654df6c6291"
format_stamp: "Formatted at 2026-09-28 08:40:30 on dist-test-slave-w6c0"
server_key: "bd3ce6e53194857d497e8aa7c67a07a3"
server_key_iv: "0be4080e633e804677b17c317fc065f1"
server_key_version: "encryptionkey@0"
I20260928 08:40:30.558501 14472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:30.581809 14472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:30.585907 14472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:30.586205 14472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:30.587247 14472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:40:30.588587 14472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:40:30.588691 14472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:30.588846 14472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:40:30.588920 14472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:30.622351 14472 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34111
I20260928 08:40:30.622411 14599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34111 every 8 connection(s)
I20260928 08:40:30.623651 14472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:40:30.630273 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 14472
I20260928 08:40:30.630451 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:40:30.630926 388 external_mini_cluster.cc:1428] Setting key 9716cccf1bbeaf576354a08dec502d89
Sep 28 08:40:30 dist-test-slave-w6c0 krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:30.645699 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:30.626267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41089 (local address 127.0.97.62:37999)
0928 08:40:30.626470 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:30.626475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:30.627683 (+ 1208us) server_negotiation.cc:407] Connection header received
0928 08:40:30.629531 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:30.629537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:30.629640 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:30.629754 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:30.631879 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:30.632718 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:30.634332 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:30.635123 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:30.635273 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:30.638783 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:30.638792 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:30.638800 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:30.638848 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:30.641970 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:30.643000 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:30.643004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:30.643006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:30.643134 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:30.643761 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:30.643766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:30.643768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:30.644085 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:30.644226 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:30.645131 (+ 905us) server_negotiation.cc:299] Negotiation successful
0928 08:40:30.645314 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 08:40:30.647382 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:30.626797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:41089)
0928 08:40:30.627511 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:30.627552 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:30.629257 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:30.629999 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:30.630008 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:30.630667 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:30.631309 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:30.631334 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:30.632881 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:30.632886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:30.634171 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:30.634184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:30.635340 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:30.635345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:30.635542 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:30.636247 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:30.636272 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:30.638475 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:30.642125 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:30.642138 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:30.642160 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:30.642867 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:30.643288 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:30.643295 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:30.643302 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:30.643569 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:30.644227 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:30.644237 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:30.644723 (+ 486us) client_negotiation.cc:769] Sending connection context
0928 08:40:30.645022 (+ 299us) client_negotiation.cc:240] Negotiation successful
0928 08:40:30.645728 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":274,"threads_started":1}
I20260928 08:40:30.649055 14600 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37999
I20260928 08:40:30.649369 14600 heartbeater.cc:464] Registering TS with master...
I20260928 08:40:30.650126 14600 heartbeater.cc:511] Master 127.0.97.62:37999 requested a full tablet report, sending...
I20260928 08:40:30.651738 14277 ts_manager.cc:194] Registered new tserver with Master: 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111)
I20260928 08:40:30.653021 14277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41089
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:40:30.685433 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37999
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:40:31.012202 14605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:31.012486 14605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:31.012586 14605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:31.020517 14605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:31.020629 14605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:31.020743 14605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:40:31.030606 14605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:31.031910 14605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:31.033176 14605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:31.046654 14610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:31.046690 14613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:31.046654 14611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:31.047729 14605 server_base.cc:1034] running on GCE node
I20260928 08:40:31.048384 14605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:31.049371 14605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:31.050738 14605 hybrid_clock.cc:648] HybridClock initialized: now 1790584831050700 us; error 47 us; skew 500 ppm
I20260928 08:40:31.051064 14605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:31 dist-test-slave-w6c0 krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:31.054265 14605 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:40:31.055751 14605 webserver.cc:460] Webserver started at http://127.0.97.3:40473/ using document root <none> and password file <none>
I20260928 08:40:31.056524 14605 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:31.056681 14605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:31.056938 14605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:40:31.059844 14605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd"
format_stamp: "Formatted at 2026-09-28 08:40:31 on dist-test-slave-w6c0"
server_key: "f8f68c9175e7e39e8e76d3fefdcc39fb"
server_key_iv: "6314816a4df4a0828fa5bb528081478f"
server_key_version: "encryptionkey@0"
I20260928 08:40:31.060632 14605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd"
format_stamp: "Formatted at 2026-09-28 08:40:31 on dist-test-slave-w6c0"
server_key: "f8f68c9175e7e39e8e76d3fefdcc39fb"
server_key_iv: "6314816a4df4a0828fa5bb528081478f"
server_key_version: "encryptionkey@0"
I20260928 08:40:31.065848 14605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 08:40:31.069340 14620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:31.070758 14605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 08:40:31.070925 14605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd"
format_stamp: "Formatted at 2026-09-28 08:40:31 on dist-test-slave-w6c0"
server_key: "f8f68c9175e7e39e8e76d3fefdcc39fb"
server_key_iv: "6314816a4df4a0828fa5bb528081478f"
server_key_version: "encryptionkey@0"
I20260928 08:40:31.071116 14605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:31.090962 14605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:31.095386 14605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:31.095681 14605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:31.096673 14605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:40:31.098093 14605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:40:31.098163 14605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:31.098348 14605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:40:31.098402 14605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:31.132030 14605 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41109
I20260928 08:40:31.132118 14732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41109 every 8 connection(s)
I20260928 08:40:31.133279 14605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:40:31.138245 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 14605
I20260928 08:40:31.138439 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:40:31.139014 388 external_mini_cluster.cc:1428] Setting key d2dca6bb5fcdc9b4a45cf9d4d7e613d1
I20260928 08:40:31.141112 14466 heartbeater.cc:503] Master 127.0.97.62:37999 was elected leader, sending a full tablet report...
Sep 28 08:40:31 dist-test-slave-w6c0 krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:31.154858 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:31.135734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39277 (local address 127.0.97.62:37999)
0928 08:40:31.135919 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:31.135923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:31.137068 (+ 1145us) server_negotiation.cc:407] Connection header received
0928 08:40:31.139563 (+ 2495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:31.139569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:31.139678 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:31.139800 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:31.141964 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.143098 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.144787 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.145365 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.145476 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:31.148866 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:31.148889 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:31.148892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:31.148923 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:31.151603 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:31.152295 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:31.152299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:31.152301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:31.152438 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:31.152893 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:31.152896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:31.152898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:31.153262 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:31.153451 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:31.154284 (+ 833us) server_negotiation.cc:299] Negotiation successful
0928 08:40:31.154543 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:40:31.155858 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:31.136144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:39277)
0928 08:40:31.136900 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:31.136942 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:31.139190 (+ 2248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:31.140098 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:31.140113 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:31.140955 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:31.141778 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:31.141798 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.143236 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.143240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:31.144638 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:31.144651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.145571 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.145576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:31.145772 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:31.146391 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:31.146418 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:31.148656 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:31.151736 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:31.151743 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:31.151755 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:31.152179 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:31.152567 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:31.152570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:31.152572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:31.152772 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:31.153431 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:31.153438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:31.153835 (+ 397us) client_negotiation.cc:769] Sending connection context
0928 08:40:31.154183 (+ 348us) client_negotiation.cc:240] Negotiation successful
0928 08:40:31.154735 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":281,"threads_started":1}
I20260928 08:40:31.157243 14733 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37999
I20260928 08:40:31.157572 14733 heartbeater.cc:464] Registering TS with master...
I20260928 08:40:31.158280 14733 heartbeater.cc:511] Master 127.0.97.62:37999 requested a full tablet report, sending...
I20260928 08:40:31.159749 14276 ts_manager.cc:194] Registered new tserver with Master: d3d51bb9dde943958dca4a3d3a5cf1fd (127.0.97.3:41109)
I20260928 08:40:31.160619 14276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39277
I20260928 08:40:31.166868 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:40:31.183135 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:31.171189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59690 (local address 127.0.97.62:37999)
0928 08:40:31.171338 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:31.171342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:31.171821 (+ 479us) server_negotiation.cc:407] Connection header received
0928 08:40:31.171954 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:31.171958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:31.172018 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:31.172097 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:31.173354 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.174070 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.175514 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.175965 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.176058 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:31.178409 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:31.178414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:31.178417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:31.178445 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:31.180890 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:31.181490 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:31.181494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:31.181496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:31.181612 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:31.181943 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:31.181945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:31.181947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:31.182230 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:31.182411 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:31.182749 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 08:40:31.182909 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 08:40:31.187067 14276 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:40:31.194373 14276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:40:31.225792 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:31.211989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.0.97.2:34111)
0928 08:40:31.212489 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:31.212498 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:31.212737 (+ 239us) server_negotiation.cc:407] Connection header received
0928 08:40:31.213052 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:31.213060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:31.213287 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:31.213494 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:31.216457 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.219234 (+ 2777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.223156 (+ 3922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.224621 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.224762 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:31.224948 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:31.225310 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 08:40:31.225491 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":237,"threads_started":1}
I20260928 08:40:31.225880 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:31.211880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34111 (local address 127.0.0.1:43528)
0928 08:40:31.212560 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:31.212618 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:31.212824 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:31.215701 (+ 2877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:31.215705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:31.215735 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:31.216235 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:31.216246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.219407 (+ 3161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.219413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:31.222974 (+ 3561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:31.222983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.224848 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.224854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:31.225034 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:31.225085 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 08:40:31.225219 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 08:40:31.225429 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":40,"thread_start_us":293,"threads_started":1}
I20260928 08:40:31.226379 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:31.213485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41109 (local address 127.0.0.1:42172)
0928 08:40:31.214039 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:31.214056 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:31.214195 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:31.215414 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:31.215422 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:31.215453 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:31.215965 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:31.215984 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.219017 (+ 3033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.219021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:31.222951 (+ 3930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:31.222971 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.225150 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.225158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:31.225355 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:31.225389 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 08:40:31.225469 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 08:40:31.225598 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":119,"thread_start_us":894,"threads_started":1}
I20260928 08:40:31.227360 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:31.214060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42172 (local address 127.0.97.3:41109)
0928 08:40:31.214774 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:31.214782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:31.214814 (+ 32us) server_negotiation.cc:407] Connection header received
0928 08:40:31.214913 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:31.214919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:31.215153 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:31.216230 (+ 1077us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:31.216527 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.218152 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.223157 (+ 5005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.224621 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.224762 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:31.224987 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:31.226881 (+ 1894us) server_negotiation.cc:299] Negotiation successful
0928 08:40:31.227042 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"spinlock_wait_cycles":18816,"thread_start_us":479,"threads_started":1}
I20260928 08:40:31.230044 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:31.214028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33225 (local address 127.0.0.1:39370)
0928 08:40:31.218685 (+ 4657us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:31.218702 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:31.218804 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:31.220375 (+ 1571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:31.220379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:31.220402 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:31.221965 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:31.221983 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.223829 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.223834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:31.225412 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:31.225427 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.229091 (+ 3664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.229097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:31.229319 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:31.229366 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 08:40:31.229462 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 08:40:31.229631 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4543,"mutex_wait_us":28,"thread_start_us":4382,"threads_started":1}
I20260928 08:40:31.230134 14535 tablet_service.cc:1467] Processing CreateTablet for tablet 0e835762b999475688205f2ae43a7471 (DEFAULT_TABLE table=test-table [id=db5938324b7f441ea3a1cac8ebdfa61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:40:31.230870 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:31.214111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39370 (local address 127.0.97.1:33225)
0928 08:40:31.219557 (+ 5446us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:31.219565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:31.219593 (+ 28us) server_negotiation.cc:407] Connection header received
0928 08:40:31.219686 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:31.219693 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:31.219909 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:31.220124 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:31.222198 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.223692 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.225580 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.228491 (+ 2911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.228652 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:31.228934 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:31.230276 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0928 08:40:31.230456 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5268,"thread_start_us":5110,"threads_started":1}
I20260928 08:40:31.230880 14668 tablet_service.cc:1467] Processing CreateTablet for tablet 0e835762b999475688205f2ae43a7471 (DEFAULT_TABLE table=test-table [id=db5938324b7f441ea3a1cac8ebdfa61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:40:31.231662 14535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e835762b999475688205f2ae43a7471. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:40:31.232164 14668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e835762b999475688205f2ae43a7471. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:40:31.238107 14401 tablet_service.cc:1467] Processing CreateTablet for tablet 0e835762b999475688205f2ae43a7471 (DEFAULT_TABLE table=test-table [id=db5938324b7f441ea3a1cac8ebdfa61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:40:31.239223 14401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e835762b999475688205f2ae43a7471. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:40:31.246934 14751 tablet_bootstrap.cc:492] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291: Bootstrap starting.
I20260928 08:40:31.251437 14751 tablet_bootstrap.cc:654] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291: Neither blocks nor log segments found. Creating new log.
I20260928 08:40:31.252785 14753 tablet_bootstrap.cc:492] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e: Bootstrap starting.
I20260928 08:40:31.252794 14751 log.cc:824] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:31.253657 14752 tablet_bootstrap.cc:492] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd: Bootstrap starting.
I20260928 08:40:31.255987 14751 tablet_bootstrap.cc:492] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291: No bootstrap required, opened a new log
I20260928 08:40:31.256286 14751 ts_tablet_manager.cc:1397] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 08:40:31.256376 14753 tablet_bootstrap.cc:654] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e: Neither blocks nor log segments found. Creating new log.
I20260928 08:40:31.257591 14753 log.cc:824] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:31.257782 14752 tablet_bootstrap.cc:654] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd: Neither blocks nor log segments found. Creating new log.
I20260928 08:40:31.258949 14752 log.cc:824] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:31.260692 14753 tablet_bootstrap.cc:492] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e: No bootstrap required, opened a new log
I20260928 08:40:31.261054 14753 ts_tablet_manager.cc:1397] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 08:40:31.262032 14752 tablet_bootstrap.cc:492] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd: No bootstrap required, opened a new log
I20260928 08:40:31.262296 14752 ts_tablet_manager.cc:1397] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 08:40:31.265148 14751 raft_consensus.cc:348] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } } peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } }
I20260928 08:40:31.265519 14751 raft_consensus.cc:374] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:40:31.265621 14751 raft_consensus.cc:729] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1374b04d764dc291f07654df6c6291, State: Initialized, Role: FOLLOWER
I20260928 08:40:31.266247 14751 consensus_queue.cc:260] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } } peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } }
I20260928 08:40:31.267357 14600 heartbeater.cc:503] Master 127.0.97.62:37999 was elected leader, sending a full tablet report...
I20260928 08:40:31.269380 14751 ts_tablet_manager.cc:1428] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260928 08:40:31.273104 14753 raft_consensus.cc:348] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } } peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } }
I20260928 08:40:31.273097 14752 raft_consensus.cc:348] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } } peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } }
I20260928 08:40:31.273536 14753 raft_consensus.cc:374] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:40:31.273536 14752 raft_consensus.cc:374] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:40:31.273641 14752 raft_consensus.cc:729] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d51bb9dde943958dca4a3d3a5cf1fd, State: Initialized, Role: FOLLOWER
I20260928 08:40:31.273641 14753 raft_consensus.cc:729] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b41b45ec8a4756acef459e9b67168e, State: Initialized, Role: FOLLOWER
I20260928 08:40:31.274257 14752 consensus_queue.cc:260] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } } peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } }
I20260928 08:40:31.274257 14753 consensus_queue.cc:260] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } } peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } }
I20260928 08:40:31.275494 14733 heartbeater.cc:503] Master 127.0.97.62:37999 was elected leader, sending a full tablet report...
I20260928 08:40:31.276301 14752 ts_tablet_manager.cc:1428] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260928 08:40:31.277093 14753 ts_tablet_manager.cc:1428] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260928 08:40:31.361701 14467 tablet.cc:2369] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:40:31.375682 14601 tablet.cc:2369] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:40:31.384943 14734 tablet.cc:2369] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:40:31.598618 14758 raft_consensus.cc:482] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:40:31.599001 14758 raft_consensus.cc:504] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } } peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } }
I20260928 08:40:31.600790 14758 leader_election.cc:288] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111), 30b41b45ec8a4756acef459e9b67168e (127.0.97.1:33225)
I20260928 08:40:31.606756 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:31.601544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56543 (local address 127.0.97.2:34111)
0928 08:40:31.601741 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:31.601747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:31.601766 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:40:31.601843 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:31.601847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:31.601934 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:31.602031 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:31.602585 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.603294 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.604982 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.605762 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.605917 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:31.606015 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:31.606299 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 08:40:31.606432 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 08:40:31.606752 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:31.600987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34111 (local address 127.0.97.3:56543)
0928 08:40:31.601213 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:31.601239 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:31.601373 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:31.602105 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:31.602111 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:31.602143 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:31.602393 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:31.602406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.603443 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.603448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:31.604823 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:31.604835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.605934 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.605940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:31.606125 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:31.606150 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 08:40:31.606263 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 08:40:31.606409 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":3}
I20260928 08:40:31.607126 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:31.601411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50453 (local address 127.0.97.1:33225)
0928 08:40:31.601624 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:31.601631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:31.601975 (+ 344us) server_negotiation.cc:407] Connection header received
0928 08:40:31.602149 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:31.602154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:31.602231 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:31.602335 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:31.602926 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.603616 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.605362 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.606153 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.606315 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:31.606420 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:31.606748 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 08:40:31.606883 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:40:31.607252 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:31.601290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33225 (local address 127.0.97.3:50453)
0928 08:40:31.601874 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:31.601891 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:31.602022 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:31.602386 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:31.602389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:31.602409 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:31.602766 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:31.602775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.603785 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.603789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:31.605194 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:31.605206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.606362 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.606368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:31.606594 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:31.606617 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 08:40:31.606720 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 08:40:31.606848 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":25,"thread_start_us":814,"threads_started":1}
I20260928 08:40:31.608147 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e835762b999475688205f2ae43a7471" candidate_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1374b04d764dc291f07654df6c6291" is_pre_election: true
I20260928 08:40:31.608147 14421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e835762b999475688205f2ae43a7471" candidate_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30b41b45ec8a4756acef459e9b67168e" is_pre_election: true
I20260928 08:40:31.608747 14555 raft_consensus.cc:2454] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3d51bb9dde943958dca4a3d3a5cf1fd in term 0.
I20260928 08:40:31.608747 14421 raft_consensus.cc:2454] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3d51bb9dde943958dca4a3d3a5cf1fd in term 0.
I20260928 08:40:31.609511 14623 leader_election.cc:302] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 30b41b45ec8a4756acef459e9b67168e, 5e1374b04d764dc291f07654df6c6291, d3d51bb9dde943958dca4a3d3a5cf1fd; no voters:
I20260928 08:40:31.609972 14758 raft_consensus.cc:2781] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:40:31.610136 14758 raft_consensus.cc:482] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:40:31.610249 14758 raft_consensus.cc:3037] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:40:31.612527 14758 raft_consensus.cc:504] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } } peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } }
I20260928 08:40:31.613117 14758 leader_election.cc:288] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [CANDIDATE]: Term 1 election: Requested vote from peers 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111), 30b41b45ec8a4756acef459e9b67168e (127.0.97.1:33225)
I20260928 08:40:31.613622 14421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e835762b999475688205f2ae43a7471" candidate_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30b41b45ec8a4756acef459e9b67168e"
I20260928 08:40:31.613620 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e835762b999475688205f2ae43a7471" candidate_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1374b04d764dc291f07654df6c6291"
I20260928 08:40:31.613838 14555 raft_consensus.cc:3037] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:40:31.613838 14421 raft_consensus.cc:3037] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:40:31.617038 14555 raft_consensus.cc:2454] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3d51bb9dde943958dca4a3d3a5cf1fd in term 1.
I20260928 08:40:31.617038 14421 raft_consensus.cc:2454] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3d51bb9dde943958dca4a3d3a5cf1fd in term 1.
I20260928 08:40:31.617499 14621 leader_election.cc:302] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 30b41b45ec8a4756acef459e9b67168e, 5e1374b04d764dc291f07654df6c6291, d3d51bb9dde943958dca4a3d3a5cf1fd; no voters:
I20260928 08:40:31.617869 14758 raft_consensus.cc:2781] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:40:31.618321 14758 raft_consensus.cc:686] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 1 LEADER]: Becoming Leader. State: Replica: d3d51bb9dde943958dca4a3d3a5cf1fd, State: Running, Role: LEADER
I20260928 08:40:31.618881 14758 consensus_queue.cc:237] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } } peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } }
I20260928 08:40:31.623679 14276 catalog_manager.cc:5495] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd reported cstate change: term changed from 0 to 1, leader changed from <none> to d3d51bb9dde943958dca4a3d3a5cf1fd (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } health_report { overall_health: UNKNOWN } } }
I20260928 08:40:31.678542 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:31.672889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.0.97.3:41109)
0928 08:40:31.673105 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:31.673112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:31.673133 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:40:31.673289 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:31.673293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:31.673381 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:31.673541 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:40:31.673934 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.674736 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.676222 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.676656 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.676793 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:31.677331 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:40:31.677886 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:40:31.678056 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:31.678109 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 08:40:31.678207 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:40:31.686872 14555 raft_consensus.cc:1261] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291 [term 1 FOLLOWER]: Refusing update from remote peer d3d51bb9dde943958dca4a3d3a5cf1fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:40:31.686867 14421 raft_consensus.cc:1261] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Refusing update from remote peer d3d51bb9dde943958dca4a3d3a5cf1fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:40:31.687851 14758 consensus_queue.cc:1035] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:40:31.688160 14761 consensus_queue.cc:1035] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:40:31.700888 14765 mvcc.cc:205] Tried to move back new op lower bound from 7334235470578724864 to 7334235470314778624. Current Snapshot: MvccSnapshot[applied={T|T < 7334235470578724864}]
I20260928 08:40:31.718472 14276 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:40:31.718796 14276 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:40:31.722105 14276 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb: Sending DeleteTablet for 3 replicas of tablet 0e835762b999475688205f2ae43a7471
I20260928 08:40:31.723237 14535 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e835762b999475688205f2ae43a7471 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:40:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43528
I20260928 08:40:31.723320 14401 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e835762b999475688205f2ae43a7471 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:40:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39370
I20260928 08:40:31.723315 14668 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e835762b999475688205f2ae43a7471 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:40:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42172
I20260928 08:40:31.725044 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 14338
I20260928 08:40:31.740265 14776 tablet_replica.cc:330] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e: stopping tablet replica
I20260928 08:40:31.740856 14776 raft_consensus.cc:2229] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:40:31.741460 14776 raft_consensus.cc:2258] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:40:31.742661 14774 tablet_replica.cc:330] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291: stopping tablet replica
I20260928 08:40:31.743211 14775 tablet_replica.cc:330] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd: stopping tablet replica
I20260928 08:40:31.743237 14774 raft_consensus.cc:2229] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:40:31.743723 14775 raft_consensus.cc:2229] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:40:31.743763 14774 raft_consensus.cc:2258] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:40:31.744585 14775 raft_consensus.cc:2258] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:40:31.745424 14776 ts_tablet_manager.cc:1905] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:40:31.746611 14775 ts_tablet_manager.cc:1905] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:40:31.749343 14774 ts_tablet_manager.cc:1905] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 08:40:31 dist-test-slave-w6c0 krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:40:31.753662 14776 ts_tablet_manager.cc:1918] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:40:31.753839 14776 log.cc:1196] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/wal/wals/0e835762b999475688205f2ae43a7471
I20260928 08:40:31.754401 14776 ts_tablet_manager.cc:1939] T 0e835762b999475688205f2ae43a7471 P 30b41b45ec8a4756acef459e9b67168e: Deleting consensus metadata
I20260928 08:40:31.754463 14775 ts_tablet_manager.cc:1918] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:40:31.754660 14775 log.cc:1196] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/0e835762b999475688205f2ae43a7471
I20260928 08:40:31.755195 14775 ts_tablet_manager.cc:1939] T 0e835762b999475688205f2ae43a7471 P d3d51bb9dde943958dca4a3d3a5cf1fd: Deleting consensus metadata
I20260928 08:40:31.756480 14263 catalog_manager.cc:4841] TS 30b41b45ec8a4756acef459e9b67168e (127.0.97.1:33225): tablet 0e835762b999475688205f2ae43a7471 (table test-table [id=db5938324b7f441ea3a1cac8ebdfa61b]) successfully deleted
I20260928 08:40:31.756796 14261 catalog_manager.cc:4841] TS d3d51bb9dde943958dca4a3d3a5cf1fd (127.0.97.3:41109): tablet 0e835762b999475688205f2ae43a7471 (table test-table [id=db5938324b7f441ea3a1cac8ebdfa61b]) successfully deleted
I20260928 08:40:31.760797 14774 ts_tablet_manager.cc:1918] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:40:31.761030 14774 log.cc:1196] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/0e835762b999475688205f2ae43a7471
I20260928 08:40:31.761596 14774 ts_tablet_manager.cc:1939] T 0e835762b999475688205f2ae43a7471 P 5e1374b04d764dc291f07654df6c6291: Deleting consensus metadata
I20260928 08:40:31.763693 14261 catalog_manager.cc:4841] TS 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111): tablet 0e835762b999475688205f2ae43a7471 (table test-table [id=db5938324b7f441ea3a1cac8ebdfa61b]) successfully deleted
I20260928 08:40:31.765066 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:31.726542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39386 (local address 127.0.97.1:33225)
0928 08:40:31.726817 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:31.726826 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:31.733163 (+ 6337us) server_negotiation.cc:407] Connection header received
0928 08:40:31.734226 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:31.734235 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:31.734373 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:31.734469 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:31.739149 (+ 4680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.740509 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.744974 (+ 4465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.745649 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.745766 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:31.757424 (+ 11658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:31.757438 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:31.757456 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:31.757508 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:31.761573 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:31.762736 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:31.762757 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:31.762761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:31.762980 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:31.763501 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:31.763506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:31.763511 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:31.764001 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:31.764302 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:31.764604 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 08:40:31.764767 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 08:40:31.765803 14461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:31.856766 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 14338
I20260928 08:40:31.875653 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 14472
Sep 28 08:40:31 dist-test-slave-w6c0 krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:40:31.891942 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:31.876445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.0.97.2:34111)
0928 08:40:31.876654 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:31.876661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:31.876683 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:40:31.876861 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:31.876866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:31.876948 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:31.877066 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:31.878238 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.879398 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.880699 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:31.881340 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:31.881463 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:31.886085 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:31.886100 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:31.886117 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:31.886152 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:31.889057 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:31.889847 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:31.889853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:31.889855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:31.889981 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:31.890363 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:31.890366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:31.890367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:31.890864 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:31.891091 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:31.891448 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 08:40:31.891609 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:40:31.892607 14595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:31.984040 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 14472
I20260928 08:40:32.008560 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 14605
Sep 28 08:40:32 dist-test-slave-w6c0 krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:40:32.027055 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:32.009513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.0.97.3:41109)
0928 08:40:32.009691 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:32.009695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:32.009715 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:40:32.009775 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:32.009779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:32.009851 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:32.009938 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:32.011783 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:32.012637 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:32.014567 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:32.015106 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:32.015210 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:32.020890 (+ 5680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:32.020902 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:32.020918 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:32.020962 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:32.024064 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:32.024912 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:32.024919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:32.024921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:32.025064 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:32.025528 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:32.025531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:32.025533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:32.025883 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:32.026100 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:32.026471 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 08:40:32.026712 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:40:32.027930 14728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:32.150730 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 14605
I20260928 08:40:32.170992 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 14245
I20260928 08:40:32.172091 14307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:32.284948 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 14245
I20260928 08:40:32.309470 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37999
--webserver_interface=127.0.97.62
--webserver_port=43905
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:40:32.635748 14782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:32.636098 14782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:32.636282 14782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:32.643443 14782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:40:32.643606 14782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:32.643726 14782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:32.643833 14782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:40:32.643922 14782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:40:32.653648 14782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:32.655050 14782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:32.656342 14782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:32.679140 14789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:32.679217 14791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:32.679164 14788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:32.680377 14782 server_base.cc:1034] running on GCE node
I20260928 08:40:32.681227 14782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:32.682888 14782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:32.684119 14782 hybrid_clock.cc:648] HybridClock initialized: now 1790584832684068 us; error 55 us; skew 500 ppm
I20260928 08:40:32.684510 14782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:32 dist-test-slave-w6c0 krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:32.688045 14782 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:40:32.689710 14782 webserver.cc:460] Webserver started at http://127.0.97.62:43905/ using document root <none> and password file <none>
I20260928 08:40:32.690474 14782 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:32.690689 14782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:32.696489 14782 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 08:40:32.699395 14798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:32.700716 14782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:40:32.700922 14782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "b8fa88119c1246d38924104841d045eb"
format_stamp: "Formatted at 2026-09-28 08:40:29 on dist-test-slave-w6c0"
server_key: "62a0b985cf4bfd74ab700a60117241e2"
server_key_iv: "d4dfd2a6cf931fada22ec07352b7338f"
server_key_version: "encryptionkey@0"
I20260928 08:40:32.702016 14782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:32.716251 14782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:32.720202 14782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:32.720489 14782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:32.739174 14782 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37999
I20260928 08:40:32.739214 14849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37999 every 8 connection(s)
I20260928 08:40:32.740686 14782 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:40:32.741518 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 14782
I20260928 08:40:32.742434 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33225
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37999
--webserver_port=45327
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:40:32.748229 14850 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:40:32.751638 14850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb: Bootstrap starting.
I20260928 08:40:32.771700 14850 log.cc:824] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:32.783285 14850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:40:32.783840 14850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb: Bootstrap complete.
I20260928 08:40:32.793012 14850 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fa88119c1246d38924104841d045eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37999 } }
I20260928 08:40:32.793875 14850 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8fa88119c1246d38924104841d045eb, State: Initialized, Role: FOLLOWER
I20260928 08:40:32.794581 14850 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fa88119c1246d38924104841d045eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37999 } }
I20260928 08:40:32.794853 14850 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:40:32.794981 14850 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:40:32.795171 14850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:40:32.798979 14850 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fa88119c1246d38924104841d045eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37999 } }
I20260928 08:40:32.799480 14850 leader_election.cc:302] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8fa88119c1246d38924104841d045eb; no voters:
I20260928 08:40:32.800148 14850 leader_election.cc:288] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 08:40:32.800290 14854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:40:32.801545 14854 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 2 LEADER]: Becoming Leader. State: Replica: b8fa88119c1246d38924104841d045eb, State: Running, Role: LEADER
I20260928 08:40:32.802111 14854 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fa88119c1246d38924104841d045eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37999 } }
I20260928 08:40:32.802662 14850 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:40:32.804612 14856 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8fa88119c1246d38924104841d045eb. Latest consensus state: current_term: 2 leader_uuid: "b8fa88119c1246d38924104841d045eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fa88119c1246d38924104841d045eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37999 } } }
I20260928 08:40:32.804957 14856 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [sys.catalog]: This master's current role is: LEADER
I20260928 08:40:32.804713 14855 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b8fa88119c1246d38924104841d045eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fa88119c1246d38924104841d045eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37999 } } }
I20260928 08:40:32.805104 14855 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [sys.catalog]: This master's current role is: LEADER
I20260928 08:40:32.817106 14859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:40:32.825484 14859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:40:32.827104 14859 catalog_manager.cc:1259] Loaded cluster ID: 52e5b7e5c4474079a1ea06cd935f0126
I20260928 08:40:32.827262 14859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:40:32.830338 14859 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 08:40:32.839046 14871 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 08:40:32.850945 14859 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 08:40:32.860008 14859 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:40:32.861662 14859 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb: Loaded TSK: 0
I20260928 08:40:32.863178 14859 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 08:40:33.088091 14852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:33.088372 14852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:33.088527 14852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:33.095883 14852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:33.095992 14852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:33.096159 14852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:40:33.106402 14852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45327
--tserver_master_addrs=127.0.97.62:37999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:33.107848 14852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:33.109205 14852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:33.123756 14878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:33.123912 14877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:33.124496 14880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:33.125386 14852 server_base.cc:1034] running on GCE node
I20260928 08:40:33.125991 14852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:33.127080 14852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:33.128340 14852 hybrid_clock.cc:648] HybridClock initialized: now 1790584833128289 us; error 35 us; skew 500 ppm
I20260928 08:40:33.128703 14852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:33 dist-test-slave-w6c0 krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:33.137245 14852 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:40:33.139019 14852 webserver.cc:460] Webserver started at http://127.0.97.1:45327/ using document root <none> and password file <none>
I20260928 08:40:33.139748 14852 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:33.139910 14852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:33.145478 14852 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 08:40:33.148473 14887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:33.149930 14852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:40:33.150103 14852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "30b41b45ec8a4756acef459e9b67168e"
format_stamp: "Formatted at 2026-09-28 08:40:30 on dist-test-slave-w6c0"
server_key: "124cd31eb95c3b9d135eff94a7e6bbfb"
server_key_iv: "bd06d33696e1bf12cd6c19d0146357af"
server_key_version: "encryptionkey@0"
I20260928 08:40:33.151173 14852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:33.162999 14852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:33.167310 14852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:33.167594 14852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:33.168699 14852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:40:33.170157 14852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:40:33.170288 14852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:33.170416 14852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:40:33.170506 14852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:33.206626 14852 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33225
I20260928 08:40:33.206698 14999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33225 every 8 connection(s)
I20260928 08:40:33.208276 14852 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:40:33.215499 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 14852
I20260928 08:40:33.216214 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34111
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37999
--webserver_port=38915
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:40:33 dist-test-slave-w6c0 krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:33.233281 15002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:33.211159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43467 (local address 127.0.97.62:37999)
0928 08:40:33.211882 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:33.211894 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:33.212498 (+ 604us) server_negotiation.cc:407] Connection header received
0928 08:40:33.214602 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:33.214615 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:33.214986 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:33.215440 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:33.216915 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:33.219062 (+ 2147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:33.220598 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:33.221657 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:33.221837 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:33.225955 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:33.225987 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:33.226006 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:33.226056 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:33.229008 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:33.229942 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:33.229947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:33.229952 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:33.230107 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:33.230596 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:33.230599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:33.230601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:33.231029 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:33.231163 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:33.232067 (+ 904us) server_negotiation.cc:299] Negotiation successful
0928 08:40:33.232383 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":316,"threads_started":1}
I20260928 08:40:33.233752 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:33.211413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:43467)
0928 08:40:33.212314 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:33.212357 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:33.214071 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:33.215507 (+ 1436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:33.215514 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:33.216073 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:33.216678 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:33.216695 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:33.219219 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:33.219223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:33.220433 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:33.220448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:33.221864 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:33.221869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:33.222067 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:33.222793 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:33.222813 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:33.225594 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:33.229149 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:33.229161 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:33.229178 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:33.229806 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:33.230235 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:33.230243 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:33.230249 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:33.230472 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:33.231193 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:33.231203 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:33.231612 (+ 409us) client_negotiation.cc:769] Sending connection context
0928 08:40:33.231908 (+ 296us) client_negotiation.cc:240] Negotiation successful
0928 08:40:33.232478 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":388,"threads_started":1}
I20260928 08:40:33.236891 15000 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37999
I20260928 08:40:33.237190 15000 heartbeater.cc:464] Registering TS with master...
I20260928 08:40:33.238009 15000 heartbeater.cc:511] Master 127.0.97.62:37999 requested a full tablet report, sending...
I20260928 08:40:33.240593 14815 ts_manager.cc:194] Registered new tserver with Master: 30b41b45ec8a4756acef459e9b67168e (127.0.97.1:33225)
I20260928 08:40:33.243002 14815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43467
W20260928 08:40:33.546794 15004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:33.547147 15004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:33.547245 15004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:33.554440 15004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:33.554580 15004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:33.554708 15004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:40:33.564560 15004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=38915
--tserver_master_addrs=127.0.97.62:37999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:33.565833 15004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:33.567207 15004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:33.580261 15012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:33.580312 15009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:33.580287 15010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:33.581032 15004 server_base.cc:1034] running on GCE node
I20260928 08:40:33.581625 15004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:33.582657 15004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:33.583878 15004 hybrid_clock.cc:648] HybridClock initialized: now 1790584833583819 us; error 54 us; skew 500 ppm
I20260928 08:40:33.584211 15004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:33 dist-test-slave-w6c0 krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:33.587606 15004 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:40:33.589155 15004 webserver.cc:460] Webserver started at http://127.0.97.2:38915/ using document root <none> and password file <none>
I20260928 08:40:33.589861 15004 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:33.590029 15004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:33.595407 15004 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:40:33.598076 15019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:33.599514 15004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:40:33.599697 15004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "5e1374b04d764dc291f07654df6c6291"
format_stamp: "Formatted at 2026-09-28 08:40:30 on dist-test-slave-w6c0"
server_key: "bd3ce6e53194857d497e8aa7c67a07a3"
server_key_iv: "0be4080e633e804677b17c317fc065f1"
server_key_version: "encryptionkey@0"
I20260928 08:40:33.600688 15004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:33.631939 15004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:33.636149 15004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:33.636447 15004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:33.637430 15004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:40:33.638866 15004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:40:33.638993 15004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:33.639132 15004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:40:33.639225 15004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:33.672596 15004 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34111
I20260928 08:40:33.672654 15131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34111 every 8 connection(s)
I20260928 08:40:33.674049 15004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:40:33.681869 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 15004
I20260928 08:40:33.682606 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41109
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37999
--webserver_port=40473
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:40:33 dist-test-slave-w6c0 krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:33.695432 15002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:33.676677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52369 (local address 127.0.97.62:37999)
0928 08:40:33.676896 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:33.676903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:33.678008 (+ 1105us) server_negotiation.cc:407] Connection header received
0928 08:40:33.679805 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:33.679810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:33.679881 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:33.679989 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:33.681377 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:33.682170 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:33.684723 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:33.685164 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:33.685262 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:33.689534 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:33.689540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:33.689543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:33.689577 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:33.692222 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:33.692991 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:33.692995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:33.692998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:33.693125 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:33.693574 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:33.693584 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:33.693586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:33.693850 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:33.694018 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:33.694814 (+ 796us) server_negotiation.cc:299] Negotiation successful
0928 08:40:33.695069 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:40:33.696447 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:33.677156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:52369)
0928 08:40:33.677846 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:33.677882 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:33.679534 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:33.680182 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:33.680189 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:33.680678 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:33.681211 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:33.681228 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:33.682330 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:33.682335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:33.684558 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:33.684577 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:33.685398 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:33.685407 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:33.685744 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:33.686528 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:33.686562 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:33.689294 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:33.692400 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:33.692412 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:33.692431 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:33.692845 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:33.693240 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:33.693245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:33.693249 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:33.693449 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:33.693990 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:33.693998 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:33.694354 (+ 356us) client_negotiation.cc:769] Sending connection context
0928 08:40:33.694699 (+ 345us) client_negotiation.cc:240] Negotiation successful
0928 08:40:33.695267 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":271,"threads_started":1}
I20260928 08:40:33.698115 15132 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37999
I20260928 08:40:33.698454 15132 heartbeater.cc:464] Registering TS with master...
I20260928 08:40:33.699203 15132 heartbeater.cc:511] Master 127.0.97.62:37999 requested a full tablet report, sending...
I20260928 08:40:33.700774 14815 ts_manager.cc:194] Registered new tserver with Master: 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111)
I20260928 08:40:33.702159 14815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52369
W20260928 08:40:34.012404 15135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:34.012758 15135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:34.012949 15135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:34.020325 15135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:34.020435 15135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:34.020609 15135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:40:34.030416 15135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40473
--tserver_master_addrs=127.0.97.62:37999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:34.031842 15135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:34.033125 15135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:34.046861 15140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:34.046825 15141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:34.046782 15143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:34.047398 15135 server_base.cc:1034] running on GCE node
I20260928 08:40:34.048136 15135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:34.049077 15135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:34.050289 15135 hybrid_clock.cc:648] HybridClock initialized: now 1790584834050241 us; error 40 us; skew 500 ppm
I20260928 08:40:34.050681 15135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:34 dist-test-slave-w6c0 krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:34.058902 15135 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:40:34.061484 15135 webserver.cc:460] Webserver started at http://127.0.97.3:40473/ using document root <none> and password file <none>
I20260928 08:40:34.062762 15135 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:34.062927 15135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:34.071302 15135 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:40:34.074862 15150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:34.076565 15135 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 08:40:34.076756 15135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd"
format_stamp: "Formatted at 2026-09-28 08:40:31 on dist-test-slave-w6c0"
server_key: "f8f68c9175e7e39e8e76d3fefdcc39fb"
server_key_iv: "6314816a4df4a0828fa5bb528081478f"
server_key_version: "encryptionkey@0"
I20260928 08:40:34.078161 15135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:34.110383 15135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:34.114614 15135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:34.114928 15135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:34.115908 15135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:40:34.117249 15135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:40:34.117369 15135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:34.117497 15135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:40:34.117590 15135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:34.150447 15135 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41109
I20260928 08:40:34.150557 15262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41109 every 8 connection(s)
I20260928 08:40:34.151890 15135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:40:34.158357 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 15135
Sep 28 08:40:34 dist-test-slave-w6c0 krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:34.173722 15002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.154488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55221 (local address 127.0.97.62:37999)
0928 08:40:34.154731 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:34.154738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:34.155999 (+ 1261us) server_negotiation.cc:407] Connection header received
0928 08:40:34.157814 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:34.157818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:34.157891 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:34.157988 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:34.160222 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.160987 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.162546 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.162972 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.163130 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:34.166852 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:34.166872 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:34.166878 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:34.166923 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:34.170195 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:34.171119 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:34.171123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:34.171126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:34.171253 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:34.171713 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:34.171718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:34.171722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:34.172077 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:34.172233 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:34.173100 (+ 867us) server_negotiation.cc:299] Negotiation successful
0928 08:40:34.173332 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"spinlock_wait_cycles":6016}
I20260928 08:40:34.174361 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.159394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59706 (local address 127.0.97.62:37999)
0928 08:40:34.160183 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:34.160190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:34.160223 (+ 33us) server_negotiation.cc:407] Connection header received
0928 08:40:34.160280 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:34.160283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:34.160341 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:34.160466 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:34.161951 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.162795 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.164161 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.164590 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.164720 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:34.167738 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:34.167746 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:34.167752 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:34.167806 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:34.171248 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:34.171968 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:34.171971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:34.171973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:34.172097 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:34.172492 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:34.172496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:34.172499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:34.172844 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:34.173080 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:34.173431 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 08:40:34.173636 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":555,"threads_started":1}
I20260928 08:40:34.175031 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.155055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:55221)
0928 08:40:34.155824 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:34.155870 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:34.157535 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:34.158282 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:34.158294 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:34.159074 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:34.160035 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:34.160059 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.161152 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.161158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:34.162379 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:34.162391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.163236 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.163243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:34.163481 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:34.164177 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:34.164199 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:34.166636 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:34.170331 (+ 3695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:34.170343 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:34.170363 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:34.170994 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:34.171389 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:34.171393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:34.171397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:34.171584 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:34.172213 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:34.172224 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:34.172622 (+ 398us) client_negotiation.cc:769] Sending connection context
0928 08:40:34.172932 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 08:40:34.173456 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":307,"threads_started":1}
I20260928 08:40:34.176928 15263 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37999
I20260928 08:40:34.177245 15263 heartbeater.cc:464] Registering TS with master...
I20260928 08:40:34.177881 15263 heartbeater.cc:511] Master 127.0.97.62:37999 requested a full tablet report, sending...
I20260928 08:40:34.179214 14815 ts_manager.cc:194] Registered new tserver with Master: d3d51bb9dde943958dca4a3d3a5cf1fd (127.0.97.3:41109)
I20260928 08:40:34.179988 14815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55221
I20260928 08:40:34.189210 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:40:34.195866 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.190128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59722 (local address 127.0.97.62:37999)
0928 08:40:34.190287 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:34.190291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:34.190857 (+ 566us) server_negotiation.cc:407] Connection header received
0928 08:40:34.191025 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:34.191029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:34.191184 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:34.191307 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:40:34.191723 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.192390 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.193861 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.194275 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.194372 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:34.194910 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:40:34.195416 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:40:34.195528 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:34.195608 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 08:40:34.195675 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:40:34.196751 14815 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:40:34.204248 14815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:40:34.243208 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.216970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.0.97.3:41109)
0928 08:40:34.231826 (+ 14856us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:34.231838 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:34.231998 (+ 160us) server_negotiation.cc:407] Connection header received
0928 08:40:34.232410 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:34.232418 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:34.232704 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:34.232888 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:34.234196 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.236461 (+ 2265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.240252 (+ 3791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.241734 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.241844 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:34.242107 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:34.242642 (+ 535us) server_negotiation.cc:299] Negotiation successful
0928 08:40:34.242844 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14614,"thread_start_us":15490,"threads_started":1}
I20260928 08:40:34.243422 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.216697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41109 (local address 127.0.0.1:42196)
0928 08:40:34.231819 (+ 15122us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:34.231860 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:34.232134 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:34.233189 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:34.233198 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:34.233238 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:34.233901 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:34.233919 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.236733 (+ 2814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.236742 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:34.240041 (+ 3299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:34.240058 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.242026 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.242032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:34.242343 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:34.242398 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 08:40:34.242593 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 08:40:34.242878 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14920,"mutex_wait_us":270,"thread_start_us":15676,"threads_started":1}
I20260928 08:40:34.245054 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.217041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33225 (local address 127.0.0.1:39388)
0928 08:40:34.232322 (+ 15281us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:34.232342 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:34.232717 (+ 375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:34.233637 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:34.233645 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:34.233687 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:34.234208 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:34.234220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.236794 (+ 2574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.236802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:34.241753 (+ 4951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:34.241768 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.244235 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.244243 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:34.244411 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:34.244450 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 08:40:34.244530 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 08:40:34.244665 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15162,"mutex_wait_us":831,"thread_start_us":15019,"threads_started":1}
I20260928 08:40:34.245055 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.217774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39388 (local address 127.0.97.1:33225)
0928 08:40:34.231026 (+ 13252us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:34.231041 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:34.232440 (+ 1399us) server_negotiation.cc:407] Connection header received
0928 08:40:34.232888 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:34.232894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:34.233127 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:34.233311 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:34.234452 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.236524 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.241929 (+ 5405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.243514 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.243663 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:34.243875 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:34.244578 (+ 703us) server_negotiation.cc:299] Negotiation successful
0928 08:40:34.244736 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12984,"thread_start_us":13272,"threads_started":1}
I20260928 08:40:34.246009 15000 heartbeater.cc:503] Master 127.0.97.62:37999 was elected leader, sending a full tablet report...
I20260928 08:40:34.247205 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.233134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34111 (local address 127.0.0.1:43552)
0928 08:40:34.239270 (+ 6136us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:34.239285 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:34.239450 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:34.240343 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:34.240348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:34.240376 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:34.240796 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:34.240810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.242176 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.242181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:34.244326 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:34.244338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.246367 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.246373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:34.246597 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:34.246633 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 08:40:34.246722 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 08:40:34.246839 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6041,"mutex_wait_us":29,"thread_start_us":5898,"threads_started":1}
I20260928 08:40:34.247892 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.233191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43552 (local address 127.0.97.2:34111)
0928 08:40:34.238987 (+ 5796us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:34.238999 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:34.239400 (+ 401us) server_negotiation.cc:407] Connection header received
0928 08:40:34.239637 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:34.239642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:34.239900 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:34.240091 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:34.240959 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.242004 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.244495 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.245925 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.246045 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:34.246220 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:34.247424 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0928 08:40:34.247600 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5594,"thread_start_us":5490,"threads_started":1}
I20260928 08:40:34.247660 15198 tablet_service.cc:1467] Processing CreateTablet for tablet b73ea3ff36e6485396c4b0d3bd766e47 (DEFAULT_TABLE table=test-table [id=b863ff70efe241e49d0ad42b1a06a313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:40:34.249094 15198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b73ea3ff36e6485396c4b0d3bd766e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:40:34.249249 14935 tablet_service.cc:1467] Processing CreateTablet for tablet b73ea3ff36e6485396c4b0d3bd766e47 (DEFAULT_TABLE table=test-table [id=b863ff70efe241e49d0ad42b1a06a313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:40:34.250216 14935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b73ea3ff36e6485396c4b0d3bd766e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:40:34.251235 15067 tablet_service.cc:1467] Processing CreateTablet for tablet b73ea3ff36e6485396c4b0d3bd766e47 (DEFAULT_TABLE table=test-table [id=b863ff70efe241e49d0ad42b1a06a313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:40:34.252467 15067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b73ea3ff36e6485396c4b0d3bd766e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:40:34.261849 15279 tablet_bootstrap.cc:492] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e: Bootstrap starting.
I20260928 08:40:34.265393 15280 tablet_bootstrap.cc:492] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd: Bootstrap starting.
I20260928 08:40:34.266559 15279 tablet_bootstrap.cc:654] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e: Neither blocks nor log segments found. Creating new log.
I20260928 08:40:34.267583 15281 tablet_bootstrap.cc:492] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291: Bootstrap starting.
I20260928 08:40:34.267884 15279 log.cc:824] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:34.269050 15280 tablet_bootstrap.cc:654] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd: Neither blocks nor log segments found. Creating new log.
I20260928 08:40:34.270287 15280 log.cc:824] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:34.271008 15281 tablet_bootstrap.cc:654] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291: Neither blocks nor log segments found. Creating new log.
I20260928 08:40:34.271380 15279 tablet_bootstrap.cc:492] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e: No bootstrap required, opened a new log
I20260928 08:40:34.271674 15279 ts_tablet_manager.cc:1397] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 08:40:34.272151 15281 log.cc:824] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:34.273335 15280 tablet_bootstrap.cc:492] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd: No bootstrap required, opened a new log
I20260928 08:40:34.273605 15280 ts_tablet_manager.cc:1397] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 08:40:34.274860 15281 tablet_bootstrap.cc:492] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291: No bootstrap required, opened a new log
I20260928 08:40:34.275122 15281 ts_tablet_manager.cc:1397] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260928 08:40:34.282455 15281 raft_consensus.cc:348] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } } peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } }
I20260928 08:40:34.282940 15281 raft_consensus.cc:374] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:40:34.283090 15281 raft_consensus.cc:729] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1374b04d764dc291f07654df6c6291, State: Initialized, Role: FOLLOWER
I20260928 08:40:34.282888 15279 raft_consensus.cc:348] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } } peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } }
I20260928 08:40:34.283294 15279 raft_consensus.cc:374] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:40:34.283419 15279 raft_consensus.cc:729] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b41b45ec8a4756acef459e9b67168e, State: Initialized, Role: FOLLOWER
I20260928 08:40:34.283794 15281 consensus_queue.cc:260] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } } peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } }
I20260928 08:40:34.283978 15279 consensus_queue.cc:260] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } } peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } }
I20260928 08:40:34.284183 15280 raft_consensus.cc:348] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } } peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } }
I20260928 08:40:34.284636 15280 raft_consensus.cc:374] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:40:34.284763 15280 raft_consensus.cc:729] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d51bb9dde943958dca4a3d3a5cf1fd, State: Initialized, Role: FOLLOWER
I20260928 08:40:34.285341 15132 heartbeater.cc:503] Master 127.0.97.62:37999 was elected leader, sending a full tablet report...
I20260928 08:40:34.285456 15280 consensus_queue.cc:260] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } } peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } }
I20260928 08:40:34.287387 15263 heartbeater.cc:503] Master 127.0.97.62:37999 was elected leader, sending a full tablet report...
I20260928 08:40:34.287645 15280 ts_tablet_manager.cc:1428] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260928 08:40:34.289237 15279 ts_tablet_manager.cc:1428] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260928 08:40:34.290434 15281 ts_tablet_manager.cc:1428] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291: Time spent starting tablet: real 0.015s user 0.007s sys 0.003s
W20260928 08:40:34.403587 15264 tablet.cc:2369] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:40:34.426204 15133 tablet.cc:2369] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:40:34.460662 15001 tablet.cc:2369] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:40:34.472050 15287 raft_consensus.cc:482] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:40:34.472378 15287 raft_consensus.cc:504] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } } peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } }
I20260928 08:40:34.474118 15287 leader_election.cc:288] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30b41b45ec8a4756acef459e9b67168e (127.0.97.1:33225), 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111)
I20260928 08:40:34.480319 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.474385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34111 (local address 127.0.97.3:44727)
0928 08:40:34.474612 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:34.474638 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:34.474768 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:34.475359 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:34.475364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:34.475396 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:34.475612 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:34.475626 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.476994 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.476998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:34.478386 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:34.478398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.479558 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.479564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:34.479737 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:34.479762 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 08:40:34.479871 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 08:40:34.480012 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":23}
I20260928 08:40:34.480311 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.474760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44727 (local address 127.0.97.2:34111)
0928 08:40:34.474994 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:34.475001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:34.475024 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:40:34.475106 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:34.475112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:34.475194 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:34.475294 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:34.475775 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.476840 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.478587 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.479384 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.479533 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:34.479629 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:34.479898 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 08:40:34.480038 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:40:34.480687 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.474613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47055 (local address 127.0.97.1:33225)
0928 08:40:34.474854 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:34.474864 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:34.475440 (+ 576us) server_negotiation.cc:407] Connection header received
0928 08:40:34.475714 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:34.475722 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:34.475801 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:34.475900 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:34.476348 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.477255 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.479031 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.479783 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.479942 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:34.480048 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:34.480314 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 08:40:34.480444 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 08:40:34.480793 15288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.474546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33225 (local address 127.0.97.3:47055)
0928 08:40:34.475336 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:34.475357 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:34.475542 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:34.475918 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:34.475921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:34.475941 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:34.476197 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:34.476204 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.477415 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.477422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:34.478859 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:34.478871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.479994 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.480000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:34.480156 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:34.480179 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 08:40:34.480287 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 08:40:34.480423 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":1,"thread_start_us":451,"threads_started":1}
I20260928 08:40:34.481448 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73ea3ff36e6485396c4b0d3bd766e47" candidate_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1374b04d764dc291f07654df6c6291" is_pre_election: true
I20260928 08:40:34.481772 14955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73ea3ff36e6485396c4b0d3bd766e47" candidate_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30b41b45ec8a4756acef459e9b67168e" is_pre_election: true
I20260928 08:40:34.481948 15087 raft_consensus.cc:2454] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3d51bb9dde943958dca4a3d3a5cf1fd in term 0.
I20260928 08:40:34.482254 14955 raft_consensus.cc:2454] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3d51bb9dde943958dca4a3d3a5cf1fd in term 0.
I20260928 08:40:34.482738 15151 leader_election.cc:302] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e1374b04d764dc291f07654df6c6291, d3d51bb9dde943958dca4a3d3a5cf1fd; no voters:
I20260928 08:40:34.483232 15287 raft_consensus.cc:2781] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:40:34.483393 15287 raft_consensus.cc:482] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:40:34.483528 15287 raft_consensus.cc:3037] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:40:34.485788 15287 raft_consensus.cc:504] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } } peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } }
I20260928 08:40:34.486423 15287 leader_election.cc:288] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [CANDIDATE]: Term 1 election: Requested vote from peers 30b41b45ec8a4756acef459e9b67168e (127.0.97.1:33225), 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111)
I20260928 08:40:34.486887 14955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73ea3ff36e6485396c4b0d3bd766e47" candidate_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30b41b45ec8a4756acef459e9b67168e"
I20260928 08:40:34.486896 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73ea3ff36e6485396c4b0d3bd766e47" candidate_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1374b04d764dc291f07654df6c6291"
I20260928 08:40:34.487088 14955 raft_consensus.cc:3037] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:40:34.487104 15087 raft_consensus.cc:3037] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:40:34.489200 14955 raft_consensus.cc:2454] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3d51bb9dde943958dca4a3d3a5cf1fd in term 1.
I20260928 08:40:34.489217 15087 raft_consensus.cc:2454] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3d51bb9dde943958dca4a3d3a5cf1fd in term 1.
I20260928 08:40:34.489626 15153 leader_election.cc:302] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30b41b45ec8a4756acef459e9b67168e, d3d51bb9dde943958dca4a3d3a5cf1fd; no voters:
I20260928 08:40:34.489921 15287 raft_consensus.cc:2781] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:40:34.490356 15287 raft_consensus.cc:686] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 1 LEADER]: Becoming Leader. State: Replica: d3d51bb9dde943958dca4a3d3a5cf1fd, State: Running, Role: LEADER
I20260928 08:40:34.490857 15287 consensus_queue.cc:237] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } } peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } }
I20260928 08:40:34.495596 14813 catalog_manager.cc:5495] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd reported cstate change: term changed from 0 to 1, leader changed from <none> to d3d51bb9dde943958dca4a3d3a5cf1fd (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } health_report { overall_health: UNKNOWN } } }
I20260928 08:40:34.531220 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.526345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45112 (local address 127.0.97.3:41109)
0928 08:40:34.526547 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:34.526552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:34.526568 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:40:34.526654 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:34.526658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:34.526743 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:34.526863 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:40:34.527189 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.527883 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.529254 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.529642 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.529734 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:34.530210 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:40:34.530746 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:40:34.530854 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:34.530934 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 08:40:34.531021 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:40:34.539268 15087 raft_consensus.cc:1261] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291 [term 1 FOLLOWER]: Refusing update from remote peer d3d51bb9dde943958dca4a3d3a5cf1fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:40:34.539268 14955 raft_consensus.cc:1261] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Refusing update from remote peer d3d51bb9dde943958dca4a3d3a5cf1fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:40:34.540197 15289 consensus_queue.cc:1035] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:40:34.540588 15287 consensus_queue.cc:1035] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:40:34.551647 15291 mvcc.cc:205] Tried to move back new op lower bound from 7334235482262138880 to 7334235482078121984. Current Snapshot: MvccSnapshot[applied={T|T < 7334235482262138880}]
I20260928 08:40:34.554090 15293 mvcc.cc:205] Tried to move back new op lower bound from 7334235482262138880 to 7334235482078121984. Current Snapshot: MvccSnapshot[applied={T|T < 7334235482262138880}]
I20260928 08:40:34.567487 14813 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:40:34.567806 14813 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:40:34.571748 14813 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb: Sending DeleteTablet for 3 replicas of tablet b73ea3ff36e6485396c4b0d3bd766e47
I20260928 08:40:34.572996 14935 tablet_service.cc:1514] Processing DeleteTablet for tablet b73ea3ff36e6485396c4b0d3bd766e47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:40:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39388
I20260928 08:40:34.573144 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 14852
I20260928 08:40:34.573150 15067 tablet_service.cc:1514] Processing DeleteTablet for tablet b73ea3ff36e6485396c4b0d3bd766e47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:40:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43552
I20260928 08:40:34.575071 15198 tablet_service.cc:1514] Processing DeleteTablet for tablet b73ea3ff36e6485396c4b0d3bd766e47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:40:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42196
I20260928 08:40:34.584493 15302 tablet_replica.cc:330] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291: stopping tablet replica
I20260928 08:40:34.585011 15302 raft_consensus.cc:2229] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:40:34.585620 15302 raft_consensus.cc:2258] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:40:34.588399 15302 ts_tablet_manager.cc:1905] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:40:34.591089 15301 tablet_replica.cc:330] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e: stopping tablet replica
I20260928 08:40:34.591657 15301 raft_consensus.cc:2229] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:40:34.592264 15301 raft_consensus.cc:2258] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:40:34.595116 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.574108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.0.97.1:33225)
0928 08:40:34.574345 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:34.574354 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:34.574380 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:40:34.574453 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:34.574461 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:34.574625 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:34.574755 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:34.577161 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.578774 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.581593 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.582593 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.582818 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:34.587637 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:34.587662 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:34.587688 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:34.587754 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:34.592068 (+ 4314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:34.592924 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:34.592932 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:34.592937 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:34.593074 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:34.593486 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:34.593492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:34.593496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:34.593864 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:34.594164 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:34.594478 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 08:40:34.594741 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 08:40:34.595963 14995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:34.599119 15303 tablet_replica.cc:330] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd: stopping tablet replica
I20260928 08:40:34.599117 15302 ts_tablet_manager.cc:1918] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:40:34.599273 15302 log.cc:1196] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/b73ea3ff36e6485396c4b0d3bd766e47
I20260928 08:40:34.599699 15303 raft_consensus.cc:2229] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:40:34.599833 15302 ts_tablet_manager.cc:1939] T b73ea3ff36e6485396c4b0d3bd766e47 P 5e1374b04d764dc291f07654df6c6291: Deleting consensus metadata
I20260928 08:40:34.600404 15303 raft_consensus.cc:2258] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:40:34.601799 14799 catalog_manager.cc:4841] TS 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111): tablet b73ea3ff36e6485396c4b0d3bd766e47 (table test-table [id=b863ff70efe241e49d0ad42b1a06a313]) successfully deleted
I20260928 08:40:34.602450 15303 ts_tablet_manager.cc:1905] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:40:34.610021 15303 ts_tablet_manager.cc:1918] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:40:34.610203 15303 log.cc:1196] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/b73ea3ff36e6485396c4b0d3bd766e47
I20260928 08:40:34.610803 15303 ts_tablet_manager.cc:1939] T b73ea3ff36e6485396c4b0d3bd766e47 P d3d51bb9dde943958dca4a3d3a5cf1fd: Deleting consensus metadata
I20260928 08:40:34.612709 14799 catalog_manager.cc:4841] TS d3d51bb9dde943958dca4a3d3a5cf1fd (127.0.97.3:41109): tablet b73ea3ff36e6485396c4b0d3bd766e47 (table test-table [id=b863ff70efe241e49d0ad42b1a06a313]) successfully deleted
I20260928 08:40:34.703413 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 14852
W20260928 08:40:34.722359 14801 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33225 (error 108)
I20260928 08:40:34.722690 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 15004
I20260928 08:40:34.724349 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.723916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33225 (local address 127.0.0.1:52752)
0928 08:40:34.724103 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:34.724207 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260928 08:40:34.724738 14801 catalog_manager.cc:4584] TS 30b41b45ec8a4756acef459e9b67168e (127.0.97.1:33225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b73ea3ff36e6485396c4b0d3bd766e47: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33225: connect: Connection refused (error 111)
I20260928 08:40:34.737828 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.723554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.0.97.2:34111)
0928 08:40:34.724123 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:34.724131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:34.724151 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:40:34.724217 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:34.724222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:34.724314 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:34.724418 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:34.725913 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.726729 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.728008 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.728422 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.728572 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:34.731416 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:34.731435 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:34.731450 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:34.731483 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:34.734322 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:34.735192 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:34.735200 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:34.735202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:34.735336 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:34.735716 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:34.735722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:34.735725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:34.736097 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:34.736888 (+ 791us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:34.737228 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 08:40:34.737454 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433}
I20260928 08:40:34.738744 15127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:34.778448 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.777894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33225 (local address 127.0.0.1:52760)
0928 08:40:34.778157 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:34.778268 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 08:40:34.779662 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.779242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33225 (local address 127.0.0.1:52768)
0928 08:40:34.779403 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:34.779505 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 08:40:34.833305 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 15004
I20260928 08:40:34.840564 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.839951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33225 (local address 127.0.0.1:52778)
0928 08:40:34.840221 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:34.840345 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 08:40:34.841913 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.841437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33225 (local address 127.0.0.1:52786)
0928 08:40:34.841640 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:34.841754 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 08:40:34.853516 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 15135
I20260928 08:40:34.867919 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.854310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45116 (local address 127.0.97.3:41109)
0928 08:40:34.854568 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:34.854575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:34.854595 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:40:34.854698 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:34.854703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:34.854778 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:34.854898 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:34.856226 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.857056 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.858357 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:34.859002 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:34.859145 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:34.861845 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:34.861857 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:34.861872 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:34.861905 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:34.864908 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:34.865602 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:34.865613 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:34.865622 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:34.865784 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:34.866175 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:34.866181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:34.866184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:34.866655 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:34.866949 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:34.867305 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 08:40:34.867468 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 08:40:34.868974 15258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:34.922956 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.922306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33225 (local address 127.0.0.1:52792)
0928 08:40:34.922610 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:34.922727 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 08:40:34.924352 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:34.923918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33225 (local address 127.0.0.1:52806)
0928 08:40:34.924140 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:34.924227 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260928 08:40:34.993623 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 15135
I20260928 08:40:35.013855 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 14782
I20260928 08:40:35.015061 14845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:35.132225 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:35.131665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33225 (local address 127.0.0.1:52818)
0928 08:40:35.131883 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:35.131986 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 08:40:35.132230 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 14782
I20260928 08:40:35.154945 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37999
--webserver_interface=127.0.97.62
--webserver_port=43905
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:40:35.489630 15309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:35.489981 15309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:35.490155 15309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:35.497344 15309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:40:35.497503 15309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:35.497614 15309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:35.497720 15309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:40:35.497810 15309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:40:35.507686 15309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:35.509115 15309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:35.510426 15309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:35.522624 15314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:35.522658 15315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:35.522583 15317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:35.523697 15309 server_base.cc:1034] running on GCE node
I20260928 08:40:35.524539 15309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:35.525957 15309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:35.527155 15309 hybrid_clock.cc:648] HybridClock initialized: now 1790584835527122 us; error 38 us; skew 500 ppm
I20260928 08:40:35.527473 15309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:35 dist-test-slave-w6c0 krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:35.530498 15309 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:40:35.531998 15309 webserver.cc:460] Webserver started at http://127.0.97.62:43905/ using document root <none> and password file <none>
I20260928 08:40:35.532706 15309 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:35.532857 15309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:35.538252 15309 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:40:35.541191 15324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:35.542572 15309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:40:35.542768 15309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "b8fa88119c1246d38924104841d045eb"
format_stamp: "Formatted at 2026-09-28 08:40:29 on dist-test-slave-w6c0"
server_key: "62a0b985cf4bfd74ab700a60117241e2"
server_key_iv: "d4dfd2a6cf931fada22ec07352b7338f"
server_key_version: "encryptionkey@0"
I20260928 08:40:35.543749 15309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:35.555370 15309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:35.559026 15309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:35.559317 15309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:35.577493 15309 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37999
I20260928 08:40:35.577523 15375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37999 every 8 connection(s)
I20260928 08:40:35.578966 15309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:40:35.585774 15376 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:40:35.588512 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 15309
I20260928 08:40:35.589566 15376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb: Bootstrap starting.
I20260928 08:40:35.589538 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33225
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37999
--webserver_port=45327
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:40:35.612354 15376 log.cc:824] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:35.628621 15376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:40:35.629190 15376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb: Bootstrap complete.
I20260928 08:40:35.638581 15376 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fa88119c1246d38924104841d045eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37999 } }
I20260928 08:40:35.639576 15376 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8fa88119c1246d38924104841d045eb, State: Initialized, Role: FOLLOWER
I20260928 08:40:35.640194 15376 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fa88119c1246d38924104841d045eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37999 } }
I20260928 08:40:35.640441 15376 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:40:35.640573 15376 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:40:35.640759 15376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 2 FOLLOWER]: Advancing to term 3
I20260928 08:40:35.644714 15376 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fa88119c1246d38924104841d045eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37999 } }
I20260928 08:40:35.645241 15376 leader_election.cc:302] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8fa88119c1246d38924104841d045eb; no voters:
I20260928 08:40:35.645932 15376 leader_election.cc:288] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 08:40:35.646070 15380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 3 FOLLOWER]: Leader election won for term 3
I20260928 08:40:35.647338 15380 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [term 3 LEADER]: Becoming Leader. State: Replica: b8fa88119c1246d38924104841d045eb, State: Running, Role: LEADER
I20260928 08:40:35.647996 15380 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fa88119c1246d38924104841d045eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37999 } }
I20260928 08:40:35.648335 15376 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:40:35.650714 15382 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8fa88119c1246d38924104841d045eb. Latest consensus state: current_term: 3 leader_uuid: "b8fa88119c1246d38924104841d045eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fa88119c1246d38924104841d045eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37999 } } }
I20260928 08:40:35.651080 15382 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [sys.catalog]: This master's current role is: LEADER
I20260928 08:40:35.651485 15381 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b8fa88119c1246d38924104841d045eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8fa88119c1246d38924104841d045eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37999 } } }
I20260928 08:40:35.651772 15381 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8fa88119c1246d38924104841d045eb [sys.catalog]: This master's current role is: LEADER
I20260928 08:40:35.652139 15387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:40:35.670925 15387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:40:35.672909 15387 catalog_manager.cc:1259] Loaded cluster ID: 52e5b7e5c4474079a1ea06cd935f0126
I20260928 08:40:35.673139 15387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 08:40:35.675982 15387 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff6c2aca9ad google::LogMessage::Fail() at ??:0
@ 0x7ff6c2acd88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff6c2aca4de google::LogMessage::Flush() at ??:0
@ 0x7ff6c2acba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff6d84fd512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff6d84fa976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
I20260928 08:40:35.710870 15399 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7ff6d8582610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff6d8582439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff6c5716345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff6c571991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff6c5719799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff6c56f4eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff6c89ce6db start_thread at ??:0
@ 0x7ff6c196461f clone at ??:0
W20260928 08:40:35.998078 15378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:35.998353 15378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:35.998449 15378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:36.005815 15378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:36.005927 15378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:36.006043 15378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:40:36.016606 15378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45327
--tserver_master_addrs=127.0.97.62:37999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:36.017841 15378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:36.019209 15378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:36.032267 15418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:36.032461 15417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:36.032438 15420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:36.032950 15378 server_base.cc:1034] running on GCE node
I20260928 08:40:36.033573 15378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:36.034447 15378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:36.035727 15378 hybrid_clock.cc:648] HybridClock initialized: now 1790584836035662 us; error 51 us; skew 500 ppm
I20260928 08:40:36.036101 15378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:36 dist-test-slave-w6c0 krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:36.039228 15378 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:40:36.040630 15378 webserver.cc:460] Webserver started at http://127.0.97.1:45327/ using document root <none> and password file <none>
I20260928 08:40:36.041353 15378 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:36.041497 15378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:36.046855 15378 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:40:36.049553 15427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:36.050875 15378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:40:36.051045 15378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "30b41b45ec8a4756acef459e9b67168e"
format_stamp: "Formatted at 2026-09-28 08:40:30 on dist-test-slave-w6c0"
server_key: "124cd31eb95c3b9d135eff94a7e6bbfb"
server_key_iv: "bd06d33696e1bf12cd6c19d0146357af"
server_key_version: "encryptionkey@0"
I20260928 08:40:36.052021 15378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:36.072455 15378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:36.076372 15378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:36.076651 15378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:36.077644 15378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:40:36.079484 15434 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260928 08:40:36.084898 15378 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 08:40:36.085037 15378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260928 08:40:36.085173 15378 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 08:40:36.088254 15378 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 08:40:36.088359 15378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260928 08:40:36.088681 15434 tablet_bootstrap.cc:492] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e: Bootstrap starting.
I20260928 08:40:36.097143 15434 log.cc:824] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:36.110489 15434 tablet_bootstrap.cc:492] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:40:36.111475 15434 tablet_bootstrap.cc:492] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e: Bootstrap complete.
I20260928 08:40:36.112053 15434 ts_tablet_manager.cc:1397] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.009s
I20260928 08:40:36.123391 15434 raft_consensus.cc:348] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } } peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } }
I20260928 08:40:36.125046 15434 raft_consensus.cc:729] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b41b45ec8a4756acef459e9b67168e, State: Initialized, Role: FOLLOWER
I20260928 08:40:36.125905 15434 consensus_queue.cc:260] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } } peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } }
I20260928 08:40:36.131625 15378 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33225
I20260928 08:40:36.131676 15542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33225 every 8 connection(s)
I20260928 08:40:36.132870 15378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:40:36.134135 15434 ts_tablet_manager.cc:1428] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e: Time spent starting tablet: real 0.022s user 0.013s sys 0.008s
I20260928 08:40:36.136888 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 15378
I20260928 08:40:36.137351 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:36.135648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:60211)
0928 08:40:36.136373 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:36.136553 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":322,"threads_started":1}
I20260928 08:40:36.137537 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34111
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37999
--webserver_port=38915
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:40:36.138144 15543 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37999 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37999: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
I20260928 08:40:36.138973 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:36.138664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:44317)
0928 08:40:36.138811 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:36.138879 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 08:40:36.139810 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:36.139477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:56013)
0928 08:40:36.139617 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:36.139700 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 08:40:36.140028 15543 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:40:36.469759 15546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:36.470057 15546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:36.470158 15546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:36.477737 15546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:36.477865 15546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:36.477983 15546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:40:36.488118 15546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=38915
--tserver_master_addrs=127.0.97.62:37999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:36.489533 15546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:36.490850 15546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:36.504560 15552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:36.504586 15551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:36.505206 15554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:36.505375 15546 server_base.cc:1034] running on GCE node
I20260928 08:40:36.506120 15546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:36.507232 15546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:36.508479 15546 hybrid_clock.cc:648] HybridClock initialized: now 1790584836508430 us; error 42 us; skew 500 ppm
I20260928 08:40:36.508814 15546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:36 dist-test-slave-w6c0 krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:36.511991 15546 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:40:36.513479 15546 webserver.cc:460] Webserver started at http://127.0.97.2:38915/ using document root <none> and password file <none>
I20260928 08:40:36.514219 15546 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:36.514374 15546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:36.519953 15546 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 08:40:36.522732 15561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:36.524085 15546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:40:36.524279 15546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "5e1374b04d764dc291f07654df6c6291"
format_stamp: "Formatted at 2026-09-28 08:40:30 on dist-test-slave-w6c0"
server_key: "bd3ce6e53194857d497e8aa7c67a07a3"
server_key_iv: "0be4080e633e804677b17c317fc065f1"
server_key_version: "encryptionkey@0"
I20260928 08:40:36.525581 15546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:36.555966 15546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:36.560322 15546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:36.560628 15546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:36.561658 15546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:40:36.563081 15546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:40:36.563201 15546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:36.563339 15546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:40:36.563431 15546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:36.597036 15546 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34111
I20260928 08:40:36.597126 15673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34111 every 8 connection(s)
I20260928 08:40:36.598485 15546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:40:36.601430 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 15546
I20260928 08:40:36.602025 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41109
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37999
--webserver_port=40473
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:40:36.603576 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:36.601591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:54563)
0928 08:40:36.602419 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:36.602711 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":338,"threads_started":1}
W20260928 08:40:36.604194 15674 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37999 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37999: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
I20260928 08:40:36.605087 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:36.604777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:35255)
0928 08:40:36.604917 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:36.604981 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 08:40:36.606034 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:36.605713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:55451)
0928 08:40:36.605850 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:36.605927 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 08:40:36.606379 15674 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:40:36.927053 15677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:36.927338 15677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:36.927435 15677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:36.934638 15677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:36.934748 15677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:36.934861 15677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:40:36.944381 15677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40473
--tserver_master_addrs=127.0.97.62:37999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:36.945629 15677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:36.946885 15677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:36.960640 15682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:36.960683 15685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:36.960722 15683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:36.961253 15677 server_base.cc:1034] running on GCE node
I20260928 08:40:36.961881 15677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:36.962920 15677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:36.964155 15677 hybrid_clock.cc:648] HybridClock initialized: now 1790584836964097 us; error 49 us; skew 500 ppm
I20260928 08:40:36.964489 15677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:36 dist-test-slave-w6c0 krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:36.967969 15677 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:40:36.969779 15677 webserver.cc:460] Webserver started at http://127.0.97.3:40473/ using document root <none> and password file <none>
I20260928 08:40:36.970724 15677 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:36.970832 15677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:36.976802 15677 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 08:40:36.980381 15692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:36.982352 15677 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 08:40:36.982631 15677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd"
format_stamp: "Formatted at 2026-09-28 08:40:31 on dist-test-slave-w6c0"
server_key: "f8f68c9175e7e39e8e76d3fefdcc39fb"
server_key_iv: "6314816a4df4a0828fa5bb528081478f"
server_key_version: "encryptionkey@0"
I20260928 08:40:36.983963 15677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:36.998287 15677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:37.002862 15677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:37.003165 15677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:37.004189 15677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:40:37.005659 15677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:40:37.005748 15677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:37.005903 15677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:40:37.006012 15677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:37.041122 15677 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41109
I20260928 08:40:37.041222 15804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41109 every 8 connection(s)
I20260928 08:40:37.042580 15677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:40:37.045223 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 15677
I20260928 08:40:37.047665 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:37.045555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:50087)
0928 08:40:37.046327 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:37.046622 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":315,"threads_started":1}
W20260928 08:40:37.048377 15805 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37999 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37999: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
I20260928 08:40:37.049176 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:37.048848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:55391)
0928 08:40:37.049000 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:37.049075 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 08:40:37.050048 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:37.049744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:47089)
0928 08:40:37.049866 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:37.049939 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 08:40:37.050308 15805 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 08:40:37.141216 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:37.140501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:44547)
0928 08:40:37.140941 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:37.141022 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":224,"threads_started":1}
I20260928 08:40:37.608029 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:37.607107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:41945)
0928 08:40:37.607691 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:37.607806 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":314,"threads_started":1}
I20260928 08:40:37.649534 15811 raft_consensus.cc:482] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:40:37.649856 15811 raft_consensus.cc:504] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } } peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } }
I20260928 08:40:37.651618 15811 leader_election.cc:288] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3d51bb9dde943958dca4a3d3a5cf1fd (127.0.97.3:41109), 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111)
Sep 28 08:40:37 dist-test-slave-w6c0 krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Sep 28 08:40:37 dist-test-slave-w6c0 krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:40:37.673379 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:37.651810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41109 (local address 127.0.97.1:47235)
0928 08:40:37.652255 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:37.652288 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:37.652827 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:37.656554 (+ 3727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:37.656584 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:37.657493 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:37.658109 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:37.658128 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:37.659698 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:37.659702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:37.661103 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:37.661115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:37.662527 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:37.662537 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:37.662736 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:37.663173 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:37.663189 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:37.666190 (+ 3001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:37.669742 (+ 3552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:37.669752 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:37.669770 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:37.670312 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:37.670805 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:37.670812 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:37.670818 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:37.671048 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:37.671818 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:37.671826 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:37.672135 (+ 309us) client_negotiation.cc:769] Sending connection context
0928 08:40:37.672403 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 08:40:37.672935 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":156,"spinlock_wait_cycles":58112,"thread_start_us":239,"threads_started":1}
I20260928 08:40:37.673367 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:37.652797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34981 (local address 127.0.97.2:34111)
0928 08:40:37.653700 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:37.653717 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:37.654080 (+ 363us) server_negotiation.cc:407] Connection header received
0928 08:40:37.655885 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:37.655894 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:37.656324 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:37.656657 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:37.658305 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:37.659506 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:37.661257 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:37.661856 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:37.662061 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:37.666822 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:37.666829 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:37.666840 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:37.666880 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:37.669872 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:37.670602 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:37.670608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:37.670612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:37.670788 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:37.671197 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:37.671206 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:37.671208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:37.671663 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:37.671866 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:37.672615 (+ 749us) server_negotiation.cc:299] Negotiation successful
0928 08:40:37.672982 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":526,"threads_started":1}
I20260928 08:40:37.673384 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:37.652150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47235 (local address 127.0.97.3:41109)
0928 08:40:37.654112 (+ 1962us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:37.654126 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:37.654164 (+ 38us) server_negotiation.cc:407] Connection header received
0928 08:40:37.655755 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:37.655765 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:37.656111 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:37.656378 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:37.658295 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:37.659512 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:37.661256 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:37.661957 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:37.662614 (+ 657us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:37.666490 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:37.666499 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:37.666544 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:37.666587 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:37.669607 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:37.670442 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:37.670451 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:37.670459 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:37.670678 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:37.671161 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:37.671166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:37.671170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:37.671654 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:37.671824 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:37.672614 (+ 790us) server_negotiation.cc:299] Negotiation successful
0928 08:40:37.672976 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1788,"thread_start_us":1684,"threads_started":1}
I20260928 08:40:37.674136 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:37.652474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34111 (local address 127.0.97.1:34981)
0928 08:40:37.653951 (+ 1477us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:37.653969 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:37.654092 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:37.656651 (+ 2559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:37.656655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:37.657494 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:37.658112 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:37.658129 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:37.659703 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:37.659708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:37.661105 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:37.661116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:37.662047 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:37.662053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:37.662307 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:37.663114 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:37.663162 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:37.666509 (+ 3347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:37.670005 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:37.670008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:37.670011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:37.670435 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:37.670899 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:37.670902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:37.670904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:37.671062 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:37.671824 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:37.671829 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:37.672131 (+ 302us) client_negotiation.cc:769] Sending connection context
0928 08:40:37.672403 (+ 272us) client_negotiation.cc:240] Negotiation successful
0928 08:40:37.673862 (+ 1459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1354,"mutex_wait_us":1220,"spinlock_wait_cycles":16256,"thread_start_us":1156,"threads_started":1}
W20260928 08:40:37.677039 15428 leader_election.cc:341] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3d51bb9dde943958dca4a3d3a5cf1fd (127.0.97.3:41109): Not found: Tablet not found: b73ea3ff36e6485396c4b0d3bd766e47
W20260928 08:40:37.677379 15428 leader_election.cc:341] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111): Not found: Tablet not found: b73ea3ff36e6485396c4b0d3bd766e47
I20260928 08:40:37.677556 15428 leader_election.cc:302] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30b41b45ec8a4756acef459e9b67168e; no voters: 5e1374b04d764dc291f07654df6c6291, d3d51bb9dde943958dca4a3d3a5cf1fd
I20260928 08:40:37.677943 15811 raft_consensus.cc:2726] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:40:38.051635 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:38.050826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:47755)
0928 08:40:38.051264 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:38.051396 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":214,"threads_started":1}
I20260928 08:40:38.142560 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:38.142095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:45155)
0928 08:40:38.142264 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:38.142366 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 08:40:38.609470 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:38.608847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:41403)
0928 08:40:38.609244 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:38.609318 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":217,"threads_started":1}
I20260928 08:40:39.053463 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:39.052667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:60227)
0928 08:40:39.053206 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:39.053304 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":274,"threads_started":1}
I20260928 08:40:39.144044 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:39.143360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:60575)
0928 08:40:39.143795 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:39.143878 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":227,"threads_started":1}
I20260928 08:40:39.594875 15820 raft_consensus.cc:482] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:40:39.595094 15820 raft_consensus.cc:504] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } } peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } }
I20260928 08:40:39.595713 15820 leader_election.cc:288] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3d51bb9dde943958dca4a3d3a5cf1fd (127.0.97.3:41109), 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111)
W20260928 08:40:39.596763 15428 leader_election.cc:341] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3d51bb9dde943958dca4a3d3a5cf1fd (127.0.97.3:41109): Not found: Tablet not found: b73ea3ff36e6485396c4b0d3bd766e47
W20260928 08:40:39.597020 15428 leader_election.cc:341] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111): Not found: Tablet not found: b73ea3ff36e6485396c4b0d3bd766e47
I20260928 08:40:39.597105 15428 leader_election.cc:302] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30b41b45ec8a4756acef459e9b67168e; no voters: 5e1374b04d764dc291f07654df6c6291, d3d51bb9dde943958dca4a3d3a5cf1fd
I20260928 08:40:39.597393 15820 raft_consensus.cc:2726] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:40:39.610934 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:39.610213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:32993)
0928 08:40:39.610686 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:39.610771 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":238,"threads_started":1}
I20260928 08:40:40.055076 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:40.054345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:59439)
0928 08:40:40.054836 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:40.054917 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":272,"threads_started":1}
I20260928 08:40:40.145741 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:40.145086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:45489)
0928 08:40:40.145494 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:40.145579 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":208,"threads_started":1}
I20260928 08:40:40.612527 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:40.611827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:49873)
0928 08:40:40.612278 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:40.612357 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":240,"threads_started":1}
I20260928 08:40:41.056710 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:41.055910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:47529)
0928 08:40:41.056436 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:41.056530 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":276,"threads_started":1}
I20260928 08:40:41.147423 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:41.146691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:32813)
0928 08:40:41.147169 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:41.147243 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":230,"threads_started":1}
I20260928 08:40:41.213532 15827 raft_consensus.cc:482] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:40:41.213763 15827 raft_consensus.cc:504] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } } peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } }
I20260928 08:40:41.214422 15827 leader_election.cc:288] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3d51bb9dde943958dca4a3d3a5cf1fd (127.0.97.3:41109), 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111)
W20260928 08:40:41.215377 15428 leader_election.cc:341] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3d51bb9dde943958dca4a3d3a5cf1fd (127.0.97.3:41109): Not found: Tablet not found: b73ea3ff36e6485396c4b0d3bd766e47
W20260928 08:40:41.215597 15428 leader_election.cc:341] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111): Not found: Tablet not found: b73ea3ff36e6485396c4b0d3bd766e47
I20260928 08:40:41.215715 15428 leader_election.cc:302] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30b41b45ec8a4756acef459e9b67168e; no voters: 5e1374b04d764dc291f07654df6c6291, d3d51bb9dde943958dca4a3d3a5cf1fd
I20260928 08:40:41.215957 15827 raft_consensus.cc:2726] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:40:41.614380 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:41.613535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:60855)
0928 08:40:41.614056 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:41.614170 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":268,"threads_started":1}
I20260928 08:40:42.058159 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:42.057523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:59655)
0928 08:40:42.057929 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:42.058003 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":212,"threads_started":1}
I20260928 08:40:42.148887 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:42.148252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:35563)
0928 08:40:42.148635 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:42.148729 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":201,"threads_started":1}
I20260928 08:40:42.615918 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:42.615236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:42559)
0928 08:40:42.615660 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:42.615740 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":220,"threads_started":1}
I20260928 08:40:43.059770 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:43.059085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:37505)
0928 08:40:43.059512 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:43.059589 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":233,"threads_started":1}
I20260928 08:40:43.150488 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:43.149751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:49795)
0928 08:40:43.150185 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:43.150284 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":233,"threads_started":1}
I20260928 08:40:43.617560 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:43.616789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:38449)
0928 08:40:43.617295 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:43.617374 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":235,"threads_started":1}
I20260928 08:40:43.717510 15835 raft_consensus.cc:482] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:40:43.717689 15835 raft_consensus.cc:504] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } } peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } }
I20260928 08:40:43.718340 15835 leader_election.cc:288] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3d51bb9dde943958dca4a3d3a5cf1fd (127.0.97.3:41109), 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111)
W20260928 08:40:43.719384 15428 leader_election.cc:341] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111): Not found: Tablet not found: b73ea3ff36e6485396c4b0d3bd766e47
W20260928 08:40:43.719606 15428 leader_election.cc:341] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3d51bb9dde943958dca4a3d3a5cf1fd (127.0.97.3:41109): Not found: Tablet not found: b73ea3ff36e6485396c4b0d3bd766e47
I20260928 08:40:43.719687 15428 leader_election.cc:302] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30b41b45ec8a4756acef459e9b67168e; no voters: 5e1374b04d764dc291f07654df6c6291, d3d51bb9dde943958dca4a3d3a5cf1fd
I20260928 08:40:43.719918 15835 raft_consensus.cc:2726] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:40:44.061510 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:44.060689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:60153)
0928 08:40:44.061233 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:44.061321 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":275,"threads_started":1}
I20260928 08:40:44.152730 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:44.151605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:60953)
0928 08:40:44.152312 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:44.152435 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":347,"threads_started":1}
I20260928 08:40:44.619217 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:44.618435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:58179)
0928 08:40:44.618946 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:44.619047 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":313,"threads_started":1}
I20260928 08:40:45.063207 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:45.062492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:54915)
0928 08:40:45.062951 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:45.063029 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":210,"threads_started":1}
I20260928 08:40:45.154572 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:45.153787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:44527)
0928 08:40:45.154219 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:45.154299 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":227,"threads_started":1}
I20260928 08:40:45.620906 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:45.620136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:40931)
0928 08:40:45.620644 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:45.620746 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":258,"threads_started":1}
I20260928 08:40:45.992936 15842 raft_consensus.cc:482] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:40:45.993108 15842 raft_consensus.cc:504] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } } peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } }
I20260928 08:40:45.993661 15842 leader_election.cc:288] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3d51bb9dde943958dca4a3d3a5cf1fd (127.0.97.3:41109), 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111)
W20260928 08:40:45.994563 15428 leader_election.cc:341] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3d51bb9dde943958dca4a3d3a5cf1fd (127.0.97.3:41109): Not found: Tablet not found: b73ea3ff36e6485396c4b0d3bd766e47
W20260928 08:40:45.994774 15428 leader_election.cc:341] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111): Not found: Tablet not found: b73ea3ff36e6485396c4b0d3bd766e47
I20260928 08:40:45.994885 15428 leader_election.cc:302] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30b41b45ec8a4756acef459e9b67168e; no voters: 5e1374b04d764dc291f07654df6c6291, d3d51bb9dde943958dca4a3d3a5cf1fd
I20260928 08:40:45.995149 15842 raft_consensus.cc:2726] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:40:46.064698 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:46.063997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:34011)
0928 08:40:46.064462 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:46.064535 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":235,"threads_started":1}
I20260928 08:40:46.156433 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:46.155612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:33761)
0928 08:40:46.156090 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:46.156166 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":227,"threads_started":1}
I20260928 08:40:46.622402 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:46.621702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:50601)
0928 08:40:46.622139 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:46.622224 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":238,"threads_started":1}
I20260928 08:40:47.066256 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:47.065570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:34811)
0928 08:40:47.065987 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:47.066066 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":226,"threads_started":1}
I20260928 08:40:47.158141 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:47.157287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:57403)
0928 08:40:47.157831 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:47.157920 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":312,"threads_started":1}
I20260928 08:40:47.623936 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:47.623269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:48061)
0928 08:40:47.623694 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:47.623777 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":211,"threads_started":1}
I20260928 08:40:48.068012 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:48.067186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:33143)
0928 08:40:48.067683 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:48.067804 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":223,"threads_started":1}
I20260928 08:40:48.159708 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:48.159024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:58681)
0928 08:40:48.159462 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:48.159554 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":216,"threads_started":1}
I20260928 08:40:48.625712 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:48.624974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:44331)
0928 08:40:48.625422 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:48.625509 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":234,"threads_started":1}
I20260928 08:40:49.069752 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:49.068903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:37959)
0928 08:40:49.069505 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:49.069580 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":383,"threads_started":1}
I20260928 08:40:49.161285 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:49.160512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:34587)
0928 08:40:49.161011 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:49.161106 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":253,"threads_started":1}
I20260928 08:40:49.627386 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:49.626755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:54319)
0928 08:40:49.627150 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:49.627226 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":201,"threads_started":1}
I20260928 08:40:50.071345 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:50.070653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:48377)
0928 08:40:50.071113 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:50.071186 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":239,"threads_started":1}
I20260928 08:40:50.162905 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:50.162159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:55883)
0928 08:40:50.162648 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:50.162725 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":243,"threads_started":1}
I20260928 08:40:50.628896 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:50.628207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:46757)
0928 08:40:50.628623 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:50.628698 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":221,"threads_started":1}
I20260928 08:40:51.072942 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:51.072218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:33061)
0928 08:40:51.072675 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:51.072752 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":250,"threads_started":1}
I20260928 08:40:51.164718 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:51.164003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.1:59259)
0928 08:40:51.164473 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:51.164547 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":230,"threads_started":1}
I20260928 08:40:51.587291 15860 raft_consensus.cc:482] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:40:51.587533 15860 raft_consensus.cc:504] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d51bb9dde943958dca4a3d3a5cf1fd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41109 } } peers { permanent_uuid: "30b41b45ec8a4756acef459e9b67168e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33225 } } peers { permanent_uuid: "5e1374b04d764dc291f07654df6c6291" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34111 } }
I20260928 08:40:51.588223 15860 leader_election.cc:288] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3d51bb9dde943958dca4a3d3a5cf1fd (127.0.97.3:41109), 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111)
W20260928 08:40:51.589146 15428 leader_election.cc:341] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3d51bb9dde943958dca4a3d3a5cf1fd (127.0.97.3:41109): Not found: Tablet not found: b73ea3ff36e6485396c4b0d3bd766e47
W20260928 08:40:51.589377 15428 leader_election.cc:341] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1374b04d764dc291f07654df6c6291 (127.0.97.2:34111): Not found: Tablet not found: b73ea3ff36e6485396c4b0d3bd766e47
I20260928 08:40:51.589524 15428 leader_election.cc:302] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30b41b45ec8a4756acef459e9b67168e; no voters: 5e1374b04d764dc291f07654df6c6291, d3d51bb9dde943958dca4a3d3a5cf1fd
I20260928 08:40:51.589797 15860 raft_consensus.cc:2726] T b73ea3ff36e6485396c4b0d3bd766e47 P 30b41b45ec8a4756acef459e9b67168e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:40:51.630600 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:51.629896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.2:52477)
0928 08:40:51.630343 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:51.630415 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":198,"threads_started":1}
I20260928 08:40:52.048753 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 15378
I20260928 08:40:52.069201 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:52.049711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44014 (local address 127.0.97.1:33225)
0928 08:40:52.050335 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:52.050346 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:52.050380 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:40:52.050457 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:52.050470 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:52.050747 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:52.050907 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:52.052656 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:52.053658 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:52.054989 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:52.055572 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:52.055705 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:52.063708 (+ 8003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:52.063719 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:52.063732 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:52.063768 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:52.066436 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:52.067169 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:52.067183 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:52.067186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:52.067311 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:52.067675 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:52.067681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:52.067685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:52.068006 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:52.068308 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:52.068585 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 08:40:52.068878 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":265,"threads_started":1}
I20260928 08:40:52.070425 15538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:52.074971 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:52.074224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37999 (local address 127.0.97.3:37575)
0928 08:40:52.074724 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:52.074835 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":229,"threads_started":1}
I20260928 08:40:52.159432 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 15378
I20260928 08:40:52.176626 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 15546
I20260928 08:40:52.189072 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:52.177280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54560 (local address 127.0.97.2:34111)
0928 08:40:52.177728 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:52.177732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:52.177747 (+ 15us) server_negotiation.cc:407] Connection header received
0928 08:40:52.177802 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:52.177805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:52.177863 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:52.177953 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:52.179298 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:52.180081 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:52.181245 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:52.181679 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:52.181771 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:52.184259 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:52.184266 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:52.184271 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:52.184309 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:52.186920 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:52.187497 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:52.187505 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:52.187508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:52.187639 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:52.187954 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:52.187957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:52.187959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:52.188208 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:52.188389 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:52.188643 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 08:40:52.188795 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":254,"threads_started":1}
I20260928 08:40:52.189769 15669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:52.271342 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 15546
I20260928 08:40:52.288391 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 15677
I20260928 08:40:52.301265 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:52.289248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47648 (local address 127.0.97.3:41109)
0928 08:40:52.289810 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:52.289815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:52.289832 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:40:52.289887 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:52.289890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:52.289993 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:52.290140 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:52.291500 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:52.292265 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:52.293397 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:52.293770 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:52.293854 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:52.296292 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:52.296297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:52.296300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:52.296328 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:52.299081 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:52.299667 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:52.299671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:52.299673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:52.299776 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:52.300069 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:52.300072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:52.300074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:52.300307 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:52.300493 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:52.300714 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 08:40:52.300855 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":298,"threads_started":1}
I20260928 08:40:52.301918 15800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:52.387974 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 15677
2026-09-28T08:40:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24660 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:40:52 dist-test-slave-w6c0 krb5kdc[15871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:40:52 dist-test-slave-w6c0 krb5kdc[15871](info): set up 2 sockets
Sep 28 08:40:52 dist-test-slave-w6c0 krb5kdc[15871](info): commencing operation
krb5kdc: starting...
W20260928 08:40:53.487120 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:40:53 dist-test-slave-w6c0 krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:40:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:40:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:40:53.717667 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34279
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:40:54.043370 15884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:54.043700 15884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:54.043855 15884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:54.050984 15884 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 08:40:54.051082 15884 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 08:40:54.051152 15884 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 08:40:54.051231 15884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:40:54.051321 15884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:54.051426 15884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:54.051512 15884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:40:54.051591 15884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:40:54.061527 15884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:54.062906 15884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:54.064201 15884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:54.076151 15889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:54.076243 15890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:54.076207 15892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:54.076890 15884 server_base.cc:1034] running on GCE node
I20260928 08:40:54.077725 15884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:54.079141 15884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:54.080391 15884 hybrid_clock.cc:648] HybridClock initialized: now 1790584854080319 us; error 36 us; skew 500 ppm
I20260928 08:40:54.080724 15884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:54 dist-test-slave-w6c0 krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:54.083845 15884 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:40:54.085215 15884 webserver.cc:460] Webserver started at http://127.0.97.62:43233/ using document root <none> and password file <none>
I20260928 08:40:54.085863 15884 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:54.085950 15884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:54.086196 15884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:40:54.089052 15884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "cc0e87f714db45809e15f16e4f885f6a"
format_stamp: "Formatted at 2026-09-28 08:40:54 on dist-test-slave-w6c0"
server_key: "92deef5e48a24d9199249df5e307860f"
server_key_iv: "7d840456cef3b2551634f251c35fa6c2"
server_key_version: "encryptionkey@0"
I20260928 08:40:54.089762 15884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "cc0e87f714db45809e15f16e4f885f6a"
format_stamp: "Formatted at 2026-09-28 08:40:54 on dist-test-slave-w6c0"
server_key: "92deef5e48a24d9199249df5e307860f"
server_key_iv: "7d840456cef3b2551634f251c35fa6c2"
server_key_version: "encryptionkey@0"
I20260928 08:40:54.094714 15884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 08:40:54.097959 15899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:54.099290 15884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:40:54.099424 15884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "cc0e87f714db45809e15f16e4f885f6a"
format_stamp: "Formatted at 2026-09-28 08:40:54 on dist-test-slave-w6c0"
server_key: "92deef5e48a24d9199249df5e307860f"
server_key_iv: "7d840456cef3b2551634f251c35fa6c2"
server_key_version: "encryptionkey@0"
I20260928 08:40:54.099596 15884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:54.119266 15884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:54.123080 15884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:54.123293 15884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:54.140712 15884 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34279
I20260928 08:40:54.140725 15950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34279 every 8 connection(s)
I20260928 08:40:54.142086 15884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:40:54.145754 15951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:40:54.148903 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 15884
I20260928 08:40:54.149098 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:40:54.149771 388 external_mini_cluster.cc:1428] Setting key b8f4c574628867bbb30eb7dfc92dac25
I20260928 08:40:54.158033 15951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a: Bootstrap starting.
I20260928 08:40:54.163414 15951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a: Neither blocks nor log segments found. Creating new log.
Sep 28 08:40:54 dist-test-slave-w6c0 krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:54.165251 15951 log.cc:824] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:54.169656 15951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a: No bootstrap required, opened a new log
I20260928 08:40:54.176004 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:54.151615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.0.97.62:34279)
0928 08:40:54.152410 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:54.152421 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:54.152460 (+ 39us) server_negotiation.cc:407] Connection header received
0928 08:40:54.154017 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:54.154036 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:54.154401 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:54.154775 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:54.156620 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:54.157861 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:54.159736 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:54.160722 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:54.161540 (+ 818us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:54.166714 (+ 5174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:54.166727 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:54.166750 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:54.166815 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:54.171076 (+ 4261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:54.172192 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:54.172205 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:54.172213 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:54.172401 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:54.172801 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:54.172809 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:54.172814 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:54.173329 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:54.174188 (+ 859us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:54.174520 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 08:40:54.175076 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":370,"threads_started":1}
I20260928 08:40:54.181995 15951 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0e87f714db45809e15f16e4f885f6a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34279 } }
I20260928 08:40:54.182480 15951 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:40:54.182624 15951 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc0e87f714db45809e15f16e4f885f6a, State: Initialized, Role: FOLLOWER
I20260928 08:40:54.183310 15951 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0e87f714db45809e15f16e4f885f6a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34279 } }
I20260928 08:40:54.183604 15951 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:40:54.183768 15951 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:40:54.183944 15951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:40:54.187461 15951 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0e87f714db45809e15f16e4f885f6a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34279 } }
I20260928 08:40:54.188043 15951 leader_election.cc:302] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc0e87f714db45809e15f16e4f885f6a; no voters:
I20260928 08:40:54.188688 15951 leader_election.cc:288] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:40:54.188824 15956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:40:54.189459 15956 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 1 LEADER]: Becoming Leader. State: Replica: cc0e87f714db45809e15f16e4f885f6a, State: Running, Role: LEADER
I20260928 08:40:54.189972 15956 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0e87f714db45809e15f16e4f885f6a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34279 } }
I20260928 08:40:54.191048 15951 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:40:54.193694 15958 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc0e87f714db45809e15f16e4f885f6a. Latest consensus state: current_term: 1 leader_uuid: "cc0e87f714db45809e15f16e4f885f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0e87f714db45809e15f16e4f885f6a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34279 } } }
I20260928 08:40:54.194090 15958 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [sys.catalog]: This master's current role is: LEADER
I20260928 08:40:54.195390 15957 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc0e87f714db45809e15f16e4f885f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0e87f714db45809e15f16e4f885f6a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34279 } } }
I20260928 08:40:54.196039 15957 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [sys.catalog]: This master's current role is: LEADER
I20260928 08:40:54.198474 15964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:40:54.203977 15964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:40:54.224875 15964 catalog_manager.cc:1347] Generated new cluster ID: 33aa991d51c843ec93897e69432e97ea
I20260928 08:40:54.225068 15964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:40:54.245322 15964 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:40:54.246649 15964 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:40:54.255440 15964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a: Generated new TSK 0
I20260928 08:40:54.256238 15964 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:40:54.318698 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34279
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:40:54.639752 15977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:54.640012 15977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:54.640161 15977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:54.647203 15977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:54.647333 15977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:54.647521 15977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:40:54.656970 15977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:54.658223 15977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:54.659593 15977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:54.672785 15982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:54.672770 15983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:54.672811 15985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:54.673457 15977 server_base.cc:1034] running on GCE node
I20260928 08:40:54.674093 15977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:54.675083 15977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:54.676296 15977 hybrid_clock.cc:648] HybridClock initialized: now 1790584854676243 us; error 49 us; skew 500 ppm
I20260928 08:40:54.676646 15977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:54 dist-test-slave-w6c0 krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:54.679692 15977 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:40:54.681098 15977 webserver.cc:460] Webserver started at http://127.0.97.1:33749/ using document root <none> and password file <none>
I20260928 08:40:54.681756 15977 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:54.681893 15977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:54.682209 15977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:40:54.685092 15977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "6a6185f2c3f747cd821eefd4e42d9880"
format_stamp: "Formatted at 2026-09-28 08:40:54 on dist-test-slave-w6c0"
server_key: "aa1d4bc936d8ffdef364e3eb5da6e187"
server_key_iv: "05b3147b670feb8bc4ab348d1cce1abf"
server_key_version: "encryptionkey@0"
I20260928 08:40:54.685863 15977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6a6185f2c3f747cd821eefd4e42d9880"
format_stamp: "Formatted at 2026-09-28 08:40:54 on dist-test-slave-w6c0"
server_key: "aa1d4bc936d8ffdef364e3eb5da6e187"
server_key_iv: "05b3147b670feb8bc4ab348d1cce1abf"
server_key_version: "encryptionkey@0"
I20260928 08:40:54.690865 15977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 08:40:54.694155 15992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:54.695495 15977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:40:54.695644 15977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "6a6185f2c3f747cd821eefd4e42d9880"
format_stamp: "Formatted at 2026-09-28 08:40:54 on dist-test-slave-w6c0"
server_key: "aa1d4bc936d8ffdef364e3eb5da6e187"
server_key_iv: "05b3147b670feb8bc4ab348d1cce1abf"
server_key_version: "encryptionkey@0"
I20260928 08:40:54.695848 15977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:54.729276 15977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:54.733261 15977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:54.733564 15977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:54.734591 15977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:40:54.735880 15977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:40:54.735953 15977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:54.736124 15977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:40:54.736196 15977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:54.768925 15977 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45691
I20260928 08:40:54.768972 16104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45691 every 8 connection(s)
I20260928 08:40:54.770179 15977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:40:54.771085 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 15977
I20260928 08:40:54.771304 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:40:54.772012 388 external_mini_cluster.cc:1428] Setting key 803761e31cf2d5f4d94ec9c1778ccbad
Sep 28 08:40:54 dist-test-slave-w6c0 krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:54.793715 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:54.772811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60229 (local address 127.0.97.62:34279)
0928 08:40:54.773297 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:54.773301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:54.774404 (+ 1103us) server_negotiation.cc:407] Connection header received
0928 08:40:54.777278 (+ 2874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:54.777282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:54.777352 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:54.777446 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:54.779471 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:54.780170 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:54.782694 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:54.783135 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:54.783322 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:54.786896 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:54.786905 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:54.786914 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:54.786962 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:54.790396 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:54.791146 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:54.791154 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:54.791160 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:54.791310 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:54.791712 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:54.791737 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:54.791743 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:54.792146 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:54.792291 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:54.792967 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 08:40:54.793222 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":263,"threads_started":1}
I20260928 08:40:54.794286 16108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:54.773389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:60229)
0928 08:40:54.774230 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:54.774286 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:54.777003 (+ 2717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:54.777705 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:54.777718 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:54.778481 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:54.779304 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:54.779330 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:54.780334 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:54.780342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:54.782527 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:54.782546 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:54.783387 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:54.783392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:54.783578 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:54.784277 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:54.784302 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:54.786672 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:54.790577 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:54.790583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:54.790594 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:54.791005 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:54.791417 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:54.791421 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:54.791423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:54.791592 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:54.792281 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:54.792287 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:54.792588 (+ 301us) client_negotiation.cc:769] Sending connection context
0928 08:40:54.792807 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 08:40:54.793176 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":360,"threads_started":1}
I20260928 08:40:54.796069 16105 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34279
I20260928 08:40:54.796383 16105 heartbeater.cc:464] Registering TS with master...
I20260928 08:40:54.797144 16105 heartbeater.cc:511] Master 127.0.97.62:34279 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:40:54.799459 15916 ts_manager.cc:194] Registered new tserver with Master: 6a6185f2c3f747cd821eefd4e42d9880 (127.0.97.1:45691)
I20260928 08:40:54.801029 15916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:40:54.829922 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34279
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:40:55.155809 16111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:55.156126 16111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:55.156307 16111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:55.163390 16111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:55.163538 16111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:55.163726 16111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:40:55.173357 16111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:55.174741 16111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:55.176055 16111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:55.188889 16119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:55.188917 16117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:55.189188 16116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:55.189626 16111 server_base.cc:1034] running on GCE node
I20260928 08:40:55.190256 16111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:55.191243 16111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:55.192490 16111 hybrid_clock.cc:648] HybridClock initialized: now 1790584855192445 us; error 34 us; skew 500 ppm
I20260928 08:40:55.192816 16111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:55 dist-test-slave-w6c0 krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:55.197553 16111 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:40:55.199818 16111 webserver.cc:460] Webserver started at http://127.0.97.2:41923/ using document root <none> and password file <none>
I20260928 08:40:55.200724 16111 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:55.200887 16111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:55.201284 16111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:40:55.205150 16111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "91a5b07dea91468881c22fdeb4281e05"
format_stamp: "Formatted at 2026-09-28 08:40:55 on dist-test-slave-w6c0"
server_key: "8bb41703a271c4f986871ddcfd31db31"
server_key_iv: "aec12794edd659d5a20f44817875be23"
server_key_version: "encryptionkey@0"
I20260928 08:40:55.205947 16111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "91a5b07dea91468881c22fdeb4281e05"
format_stamp: "Formatted at 2026-09-28 08:40:55 on dist-test-slave-w6c0"
server_key: "8bb41703a271c4f986871ddcfd31db31"
server_key_iv: "aec12794edd659d5a20f44817875be23"
server_key_version: "encryptionkey@0"
I20260928 08:40:55.211578 16111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:40:55.215145 16126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:55.216451 16111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:40:55.216635 16111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "91a5b07dea91468881c22fdeb4281e05"
format_stamp: "Formatted at 2026-09-28 08:40:55 on dist-test-slave-w6c0"
server_key: "8bb41703a271c4f986871ddcfd31db31"
server_key_iv: "aec12794edd659d5a20f44817875be23"
server_key_version: "encryptionkey@0"
I20260928 08:40:55.216867 16111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:55.237778 16111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:55.242014 16111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:55.242345 16111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:55.243443 16111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:40:55.244855 16111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:40:55.244976 16111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:55.245133 16111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:40:55.245208 16111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:55.278723 16111 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36155
I20260928 08:40:55.278774 16238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36155 every 8 connection(s)
I20260928 08:40:55.280037 16111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:40:55.282267 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 16111
I20260928 08:40:55.282469 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:40:55.283020 388 external_mini_cluster.cc:1428] Setting key a19e3d29885beed3acad37f6d71bf11b
Sep 28 08:40:55 dist-test-slave-w6c0 krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:55.303915 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:55.282722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50521 (local address 127.0.97.62:34279)
0928 08:40:55.282960 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:55.282966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:55.284244 (+ 1278us) server_negotiation.cc:407] Connection header received
0928 08:40:55.286739 (+ 2495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:55.286743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:55.286813 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:55.286904 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:55.288992 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.289684 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.292233 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.292676 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.292773 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:55.296645 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:55.296653 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:55.296662 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:55.296710 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:55.300466 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:55.301232 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:55.301240 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:55.301246 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:55.301407 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:55.301822 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:55.301830 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:55.301836 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:55.302227 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:55.302378 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:55.303176 (+ 798us) server_negotiation.cc:299] Negotiation successful
0928 08:40:55.303436 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 08:40:55.304682 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:55.283273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:50521)
0928 08:40:55.284078 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:55.284126 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:55.286417 (+ 2291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:55.287182 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:55.287198 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:55.288003 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:55.288811 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:55.288837 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.289849 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.289857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:55.292061 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:55.292080 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.292907 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.292916 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:55.293231 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:55.294008 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:55.294036 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:55.296402 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:55.300602 (+ 4200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:55.300609 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:55.300624 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:55.301078 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:55.301517 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:55.301520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:55.301522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:55.301691 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:55.302365 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:55.302378 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:55.302747 (+ 369us) client_negotiation.cc:769] Sending connection context
0928 08:40:55.303060 (+ 313us) client_negotiation.cc:240] Negotiation successful
0928 08:40:55.303548 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":313,"threads_started":1}
I20260928 08:40:55.306208 16239 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34279
I20260928 08:40:55.306568 16239 heartbeater.cc:464] Registering TS with master...
I20260928 08:40:55.307281 16239 heartbeater.cc:511] Master 127.0.97.62:34279 requested a full tablet report, sending...
I20260928 08:40:55.308758 15916 ts_manager.cc:194] Registered new tserver with Master: 91a5b07dea91468881c22fdeb4281e05 (127.0.97.2:36155)
I20260928 08:40:55.309580 15916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50521
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:40:55.339823 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34279
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:40:55.655288 16244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:40:55.655610 16244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:40:55.655789 16244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:40:55.663084 16244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:40:55.663239 16244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:40:55.663436 16244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:40:55.673712 16244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:40:55.675087 16244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:40:55.676415 16244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:40:55.689599 16250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:55.689636 16249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:40:55.690382 16252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:40:55.692063 16244 server_base.cc:1034] running on GCE node
I20260928 08:40:55.692700 16244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:40:55.693701 16244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:40:55.694931 16244 hybrid_clock.cc:648] HybridClock initialized: now 1790584855694894 us; error 61 us; skew 500 ppm
I20260928 08:40:55.695250 16244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:40:55 dist-test-slave-w6c0 krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:40:55.698556 16244 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:40:55.700228 16244 webserver.cc:460] Webserver started at http://127.0.97.3:38389/ using document root <none> and password file <none>
I20260928 08:40:55.701005 16244 fs_manager.cc:359] Metadata directory not provided
I20260928 08:40:55.701151 16244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:40:55.701488 16244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:40:55.704658 16244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "e491b4648bcd40bab94d37e206ebaa6a"
format_stamp: "Formatted at 2026-09-28 08:40:55 on dist-test-slave-w6c0"
server_key: "441fe1d7e9880e188ac4f5c169142826"
server_key_iv: "44d7cf3fd5639f9125a6e2f9bb40136a"
server_key_version: "encryptionkey@0"
I20260928 08:40:55.705425 16244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e491b4648bcd40bab94d37e206ebaa6a"
format_stamp: "Formatted at 2026-09-28 08:40:55 on dist-test-slave-w6c0"
server_key: "441fe1d7e9880e188ac4f5c169142826"
server_key_iv: "44d7cf3fd5639f9125a6e2f9bb40136a"
server_key_version: "encryptionkey@0"
I20260928 08:40:55.710654 16244 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 08:40:55.714118 16259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:55.715619 16244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:40:55.715801 16244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "e491b4648bcd40bab94d37e206ebaa6a"
format_stamp: "Formatted at 2026-09-28 08:40:55 on dist-test-slave-w6c0"
server_key: "441fe1d7e9880e188ac4f5c169142826"
server_key_iv: "44d7cf3fd5639f9125a6e2f9bb40136a"
server_key_version: "encryptionkey@0"
I20260928 08:40:55.716004 16244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:40:55.739120 16244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:40:55.743295 16244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:40:55.743592 16244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:40:55.744565 16244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:40:55.745916 16244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:40:55.746021 16244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:55.746172 16244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:40:55.746244 16244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:40:55.780160 16244 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45131
I20260928 08:40:55.780242 16371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45131 every 8 connection(s)
I20260928 08:40:55.781391 16244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:40:55.791126 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 16244
I20260928 08:40:55.791318 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:40:55.791760 388 external_mini_cluster.cc:1428] Setting key 6e35cbfdc3a22432a0eedfeb433e020c
Sep 28 08:40:55 dist-test-slave-w6c0 krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:40:55.800500 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:55.783916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39331 (local address 127.0.97.62:34279)
0928 08:40:55.784120 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:55.784126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:55.785010 (+ 884us) server_negotiation.cc:407] Connection header received
0928 08:40:55.786877 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:55.786883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:55.786979 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:55.787088 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:55.788460 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.789234 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.790797 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.791415 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.791567 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:55.795296 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:55.795300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:55.795303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:55.795338 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:55.797625 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:55.798444 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:55.798448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:55.798450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:55.798657 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:55.799000 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:55.799003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:55.799005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:55.799238 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:55.799345 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:55.800066 (+ 721us) server_negotiation.cc:299] Negotiation successful
0928 08:40:55.800224 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 08:40:55.801386 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:55.784177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:39331)
0928 08:40:55.784851 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:55.784891 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:55.786623 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:55.787300 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:55.787308 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:55.787793 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:40:55.788297 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:55.788312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.789384 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.789394 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:55.790647 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:55.790658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.791627 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.791637 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:55.791913 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:55.792682 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:40:55.792703 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:40:55.795133 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:40:55.797755 (+ 2622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:55.797781 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:55.797815 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:55.798325 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:55.798731 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:40:55.798734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:40:55.798736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:40:55.798907 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:40:55.799345 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:40:55.799350 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:40:55.799674 (+ 324us) client_negotiation.cc:769] Sending connection context
0928 08:40:55.799949 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 08:40:55.800342 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":278,"threads_started":1}
I20260928 08:40:55.802713 16372 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34279
I20260928 08:40:55.803025 16372 heartbeater.cc:464] Registering TS with master...
I20260928 08:40:55.803736 16372 heartbeater.cc:511] Master 127.0.97.62:34279 requested a full tablet report, sending...
I20260928 08:40:55.804718 16105 heartbeater.cc:503] Master 127.0.97.62:34279 was elected leader, sending a full tablet report...
I20260928 08:40:55.805440 15916 ts_manager.cc:194] Registered new tserver with Master: e491b4648bcd40bab94d37e206ebaa6a (127.0.97.3:45131)
I20260928 08:40:55.806581 15916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39331
I20260928 08:40:55.807485 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:40:55.823901 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:55.811753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49466 (local address 127.0.97.62:34279)
0928 08:40:55.811931 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:55.811935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:55.812309 (+ 374us) server_negotiation.cc:407] Connection header received
0928 08:40:55.812430 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:55.812433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:55.812485 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:55.812558 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:55.814000 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.814749 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.816105 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.816463 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.816548 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:55.818907 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:55.818911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:55.818914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:55.818942 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:55.821709 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:55.822318 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:55.822321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:55.822323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:55.822447 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:55.822800 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:55.822803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:55.822805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:55.823102 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:55.823278 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:55.823536 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 08:40:55.823681 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:40:55.827610 15914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:40:55.834426 15914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:40:55.858695 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:55.847737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48794 (local address 127.0.97.3:45131)
0928 08:40:55.849295 (+ 1558us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:55.849305 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:55.849344 (+ 39us) server_negotiation.cc:407] Connection header received
0928 08:40:55.849468 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:55.849478 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:55.849764 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:55.849986 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:55.850938 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.852568 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.855285 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.856702 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.856847 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:55.857064 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:55.858136 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0928 08:40:55.858308 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1360,"thread_start_us":1174,"threads_started":1}
I20260928 08:40:55.858717 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:55.846764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45131 (local address 127.0.0.1:48794)
0928 08:40:55.847873 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:55.847892 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:55.848013 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:55.850041 (+ 2028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:55.850052 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:55.850098 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:55.850711 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:55.850724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.852786 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.852792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:55.855094 (+ 2302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:55.855113 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.857643 (+ 2530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.857649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:55.857879 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:55.857941 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 08:40:55.858082 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 08:40:55.858261 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":24,"thread_start_us":966,"threads_started":1}
I20260928 08:40:55.859642 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:55.847518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34992 (local address 127.0.97.1:45691)
0928 08:40:55.848717 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:55.848729 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:55.850969 (+ 2240us) server_negotiation.cc:407] Connection header received
0928 08:40:55.851291 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:55.851302 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:55.851583 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:55.851725 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:55.853142 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.854720 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.856634 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.858100 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.858208 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:55.858412 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:55.859181 (+ 769us) server_negotiation.cc:299] Negotiation successful
0928 08:40:55.859333 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":970,"thread_start_us":752,"threads_started":1}
I20260928 08:40:55.859704 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:55.847189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45691 (local address 127.0.0.1:34992)
0928 08:40:55.850844 (+ 3655us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:55.850864 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:55.851089 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:55.851818 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:55.851824 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:55.851848 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:55.852173 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:55.852181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.854868 (+ 2687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.854875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:55.856480 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:55.856491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.858566 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.858577 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:55.858763 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:55.858800 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 08:40:55.859143 (+ 343us) client_negotiation.cc:240] Negotiation successful
0928 08:40:55.859264 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3492,"mutex_wait_us":24,"thread_start_us":3292,"threads_started":1}
I20260928 08:40:55.859725 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:55.846115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36155 (local address 127.0.0.1:50880)
0928 08:40:55.846899 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:55.846948 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:55.847168 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:55.852275 (+ 5107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:55.852281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:55.852307 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:55.852644 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:55.852659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.854933 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.854939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:55.857154 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:55.857166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.858895 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.858902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:55.859079 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:55.859121 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 08:40:55.859187 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 08:40:55.859322 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":2,"thread_start_us":274,"threads_started":1}
I20260928 08:40:55.860560 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:55.846266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50880 (local address 127.0.97.2:36155)
0928 08:40:55.851789 (+ 5523us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:55.851797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:55.851822 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:40:55.851910 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:55.851916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:55.852091 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:55.852268 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:55.853014 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.854739 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.857299 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.858659 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.859985 (+ 1326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:55.860163 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:55.860239 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 08:40:55.860365 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5355,"thread_start_us":5255,"threads_started":1}
I20260928 08:40:55.863095 16040 tablet_service.cc:1467] Processing CreateTablet for tablet 59455a9b73f4403e9f6d718957336a53 (DEFAULT_TABLE table=test-table [id=5019798c2dac40a4ac7942def465aae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:40:55.863124 16307 tablet_service.cc:1467] Processing CreateTablet for tablet 59455a9b73f4403e9f6d718957336a53 (DEFAULT_TABLE table=test-table [id=5019798c2dac40a4ac7942def465aae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:40:55.863793 16174 tablet_service.cc:1467] Processing CreateTablet for tablet 59455a9b73f4403e9f6d718957336a53 (DEFAULT_TABLE table=test-table [id=5019798c2dac40a4ac7942def465aae3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:40:55.864372 16040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59455a9b73f4403e9f6d718957336a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:40:55.864487 16307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59455a9b73f4403e9f6d718957336a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:40:55.864959 16174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59455a9b73f4403e9f6d718957336a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:40:55.875775 16390 tablet_bootstrap.cc:492] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880: Bootstrap starting.
I20260928 08:40:55.879206 16390 tablet_bootstrap.cc:654] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880: Neither blocks nor log segments found. Creating new log.
I20260928 08:40:55.879406 16392 tablet_bootstrap.cc:492] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a: Bootstrap starting.
I20260928 08:40:55.880106 16391 tablet_bootstrap.cc:492] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05: Bootstrap starting.
I20260928 08:40:55.880609 16390 log.cc:824] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:55.883559 16391 tablet_bootstrap.cc:654] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05: Neither blocks nor log segments found. Creating new log.
I20260928 08:40:55.884222 16390 tablet_bootstrap.cc:492] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880: No bootstrap required, opened a new log
I20260928 08:40:55.884474 16390 ts_tablet_manager.cc:1397] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 08:40:55.884603 16392 tablet_bootstrap.cc:654] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a: Neither blocks nor log segments found. Creating new log.
I20260928 08:40:55.884742 16391 log.cc:824] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:55.885875 16392 log.cc:824] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:40:55.892712 16390 raft_consensus.cc:348] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } }
I20260928 08:40:55.893054 16391 tablet_bootstrap.cc:492] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05: No bootstrap required, opened a new log
I20260928 08:40:55.893070 16390 raft_consensus.cc:374] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:40:55.893168 16390 raft_consensus.cc:729] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6185f2c3f747cd821eefd4e42d9880, State: Initialized, Role: FOLLOWER
I20260928 08:40:55.893337 16391 ts_tablet_manager.cc:1397] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260928 08:40:55.893759 16390 consensus_queue.cc:260] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } }
I20260928 08:40:55.893937 16392 tablet_bootstrap.cc:492] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a: No bootstrap required, opened a new log
I20260928 08:40:55.894187 16392 ts_tablet_manager.cc:1397] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260928 08:40:55.904210 16391 raft_consensus.cc:348] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } }
I20260928 08:40:55.904704 16391 raft_consensus.cc:374] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:40:55.904825 16391 raft_consensus.cc:729] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91a5b07dea91468881c22fdeb4281e05, State: Initialized, Role: FOLLOWER
I20260928 08:40:55.904772 16392 raft_consensus.cc:348] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } }
I20260928 08:40:55.905146 16392 raft_consensus.cc:374] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:40:55.905246 16392 raft_consensus.cc:729] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e491b4648bcd40bab94d37e206ebaa6a, State: Initialized, Role: FOLLOWER
I20260928 08:40:55.905923 16392 consensus_queue.cc:260] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } }
I20260928 08:40:55.906770 16390 ts_tablet_manager.cc:1428] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880: Time spent starting tablet: real 0.022s user 0.018s sys 0.005s
I20260928 08:40:55.907291 16396 raft_consensus.cc:482] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:40:55.907613 16396 raft_consensus.cc:504] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } }
I20260928 08:40:55.911360 16392 ts_tablet_manager.cc:1428] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260928 08:40:55.911795 16372 heartbeater.cc:503] Master 127.0.97.62:34279 was elected leader, sending a full tablet report...
I20260928 08:40:55.912238 16391 consensus_queue.cc:260] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } }
I20260928 08:40:55.913130 16396 leader_election.cc:288] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91a5b07dea91468881c22fdeb4281e05 (127.0.97.2:36155), e491b4648bcd40bab94d37e206ebaa6a (127.0.97.3:45131)
I20260928 08:40:55.914755 16239 heartbeater.cc:503] Master 127.0.97.62:34279 was elected leader, sending a full tablet report...
I20260928 08:40:55.915612 16391 ts_tablet_manager.cc:1428] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05: Time spent starting tablet: real 0.022s user 0.005s sys 0.009s
I20260928 08:40:55.921941 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:55.913513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58641 (local address 127.0.97.3:45131)
0928 08:40:55.913722 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:55.913731 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:55.914517 (+ 786us) server_negotiation.cc:407] Connection header received
0928 08:40:55.914765 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:55.914769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:55.914838 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:55.914916 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:55.915935 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.917042 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.919653 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.920929 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.921003 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:55.921073 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:55.921576 (+ 503us) server_negotiation.cc:299] Negotiation successful
0928 08:40:55.921693 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 08:40:55.922035 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:55.913764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45131 (local address 127.0.97.1:58641)
0928 08:40:55.914388 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:55.914413 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:55.914606 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:55.915179 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:55.915184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:55.915207 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:55.915590 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:55.915604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.917203 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.917209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:55.919472 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:55.919490 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.921284 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.921290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:55.921416 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:55.921438 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 08:40:55.921543 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 08:40:55.921667 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":2209,"threads_started":1}
I20260928 08:40:55.922711 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:55.910116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50957 (local address 127.0.97.2:36155)
0928 08:40:55.910291 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:55.910297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:55.914313 (+ 4016us) server_negotiation.cc:407] Connection header received
0928 08:40:55.914530 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:55.914539 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:55.914618 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:55.914708 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:40:55.916741 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.917735 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.920239 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.921532 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.921632 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:55.921719 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:55.922312 (+ 593us) server_negotiation.cc:299] Negotiation successful
0928 08:40:55.922436 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 08:40:55.922785 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:55.910030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36155 (local address 127.0.97.1:50957)
0928 08:40:55.914184 (+ 4154us) negotiation.cc:107] Waiting for socket to connect
0928 08:40:55.914211 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:40:55.914375 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:40:55.916145 (+ 1770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:40:55.916151 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:40:55.916177 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:40:55.916540 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:55.916552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.917940 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.917947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:55.920045 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:40:55.920072 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.921942 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.921950 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:40:55.922142 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:55.922188 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 08:40:55.922283 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 08:40:55.922407 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3992,"thread_start_us":3837,"threads_started":1}
I20260928 08:40:55.923084 16327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59455a9b73f4403e9f6d718957336a53" candidate_uuid: "6a6185f2c3f747cd821eefd4e42d9880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e491b4648bcd40bab94d37e206ebaa6a" is_pre_election: true
I20260928 08:40:55.923552 16327 raft_consensus.cc:2454] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a6185f2c3f747cd821eefd4e42d9880 in term 0.
I20260928 08:40:55.923586 16194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59455a9b73f4403e9f6d718957336a53" candidate_uuid: "6a6185f2c3f747cd821eefd4e42d9880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91a5b07dea91468881c22fdeb4281e05" is_pre_election: true
I20260928 08:40:55.924054 16194 raft_consensus.cc:2454] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a6185f2c3f747cd821eefd4e42d9880 in term 0.
I20260928 08:40:55.924209 15996 leader_election.cc:302] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a6185f2c3f747cd821eefd4e42d9880, e491b4648bcd40bab94d37e206ebaa6a; no voters:
I20260928 08:40:55.924600 16396 raft_consensus.cc:2781] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:40:55.924748 16396 raft_consensus.cc:482] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:40:55.924860 16396 raft_consensus.cc:3037] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:40:55.927063 16396 raft_consensus.cc:504] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } }
I20260928 08:40:55.927881 16396 leader_election.cc:288] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 1 election: Requested vote from peers 91a5b07dea91468881c22fdeb4281e05 (127.0.97.2:36155), e491b4648bcd40bab94d37e206ebaa6a (127.0.97.3:45131)
I20260928 08:40:55.928246 16194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59455a9b73f4403e9f6d718957336a53" candidate_uuid: "6a6185f2c3f747cd821eefd4e42d9880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91a5b07dea91468881c22fdeb4281e05"
I20260928 08:40:55.928298 16327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59455a9b73f4403e9f6d718957336a53" candidate_uuid: "6a6185f2c3f747cd821eefd4e42d9880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e491b4648bcd40bab94d37e206ebaa6a"
I20260928 08:40:55.928462 16194 raft_consensus.cc:3037] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:40:55.928481 16327 raft_consensus.cc:3037] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:40:55.930805 16327 raft_consensus.cc:2454] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a6185f2c3f747cd821eefd4e42d9880 in term 1.
I20260928 08:40:55.931195 15996 leader_election.cc:302] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a6185f2c3f747cd821eefd4e42d9880, e491b4648bcd40bab94d37e206ebaa6a; no voters:
I20260928 08:40:55.931538 16396 raft_consensus.cc:2781] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:40:55.931715 16396 raft_consensus.cc:686] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 LEADER]: Becoming Leader. State: Replica: 6a6185f2c3f747cd821eefd4e42d9880, State: Running, Role: LEADER
I20260928 08:40:55.932039 16194 raft_consensus.cc:2454] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a6185f2c3f747cd821eefd4e42d9880 in term 1.
I20260928 08:40:55.932197 16396 consensus_queue.cc:237] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } }
I20260928 08:40:55.936568 15914 catalog_manager.cc:5495] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a6185f2c3f747cd821eefd4e42d9880 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "6a6185f2c3f747cd821eefd4e42d9880" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } health_report { overall_health: UNKNOWN } } }
I20260928 08:40:55.967017 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:55.961806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35008 (local address 127.0.97.1:45691)
0928 08:40:55.961974 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:55.961979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:55.961996 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:40:55.962163 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:55.962167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:55.962241 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:55.962390 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:40:55.962799 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.963532 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.964931 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:55.965338 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:55.965430 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:55.965872 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:40:55.966369 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:40:55.966479 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:55.966643 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 08:40:55.966708 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 08:40:55.974694 16194 raft_consensus.cc:1261] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05 [term 1 FOLLOWER]: Refusing update from remote peer 6a6185f2c3f747cd821eefd4e42d9880: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:40:55.974702 16327 raft_consensus.cc:1261] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a [term 1 FOLLOWER]: Refusing update from remote peer 6a6185f2c3f747cd821eefd4e42d9880: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:40:55.975631 16397 consensus_queue.cc:1035] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:40:55.975992 16396 consensus_queue.cc:1035] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:40:56.000310 15916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:40:56.000664 15916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:40:56.004410 15916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a: Sending DeleteTablet for 3 replicas of tablet 59455a9b73f4403e9f6d718957336a53
I20260928 08:40:56.005684 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 15977
I20260928 08:40:56.005578 16307 tablet_service.cc:1514] Processing DeleteTablet for tablet 59455a9b73f4403e9f6d718957336a53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:40:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48794
I20260928 08:40:56.006557 16414 tablet_replica.cc:330] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a: stopping tablet replica
I20260928 08:40:56.006610 16040 tablet_service.cc:1514] Processing DeleteTablet for tablet 59455a9b73f4403e9f6d718957336a53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:40:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34992
I20260928 08:40:56.006379 16174 tablet_service.cc:1514] Processing DeleteTablet for tablet 59455a9b73f4403e9f6d718957336a53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:40:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50880
I20260928 08:40:56.007130 16414 raft_consensus.cc:2229] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:40:56.007718 16414 raft_consensus.cc:2258] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:40:56.009892 16414 ts_tablet_manager.cc:1905] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:40:56.017129 16414 ts_tablet_manager.cc:1918] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:40:56.017397 16414 log.cc:1196] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/59455a9b73f4403e9f6d718957336a53
I20260928 08:40:56.018010 16414 ts_tablet_manager.cc:1939] T 59455a9b73f4403e9f6d718957336a53 P e491b4648bcd40bab94d37e206ebaa6a: Deleting consensus metadata
I20260928 08:40:56.019789 16416 tablet_replica.cc:330] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880: stopping tablet replica
I20260928 08:40:56.020212 15903 catalog_manager.cc:4841] TS e491b4648bcd40bab94d37e206ebaa6a (127.0.97.3:45131): tablet 59455a9b73f4403e9f6d718957336a53 (table test-table [id=5019798c2dac40a4ac7942def465aae3]) successfully deleted
I20260928 08:40:56.020398 16416 raft_consensus.cc:2229] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:40:56.021175 16416 raft_consensus.cc:2258] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:40:56.023358 16416 ts_tablet_manager.cc:1905] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:40:56.027067 16417 tablet_replica.cc:330] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05: stopping tablet replica
I20260928 08:40:56.027550 16417 raft_consensus.cc:2229] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:40:56.028057 16417 raft_consensus.cc:2258] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:40:56.030228 16417 ts_tablet_manager.cc:1905] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:40:56.031623 16416 ts_tablet_manager.cc:1918] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:40:56.031821 16416 log.cc:1196] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/wal/wals/59455a9b73f4403e9f6d718957336a53
I20260928 08:40:56.032435 16416 ts_tablet_manager.cc:1939] T 59455a9b73f4403e9f6d718957336a53 P 6a6185f2c3f747cd821eefd4e42d9880: Deleting consensus metadata
I20260928 08:40:56.034245 15902 catalog_manager.cc:4841] TS 6a6185f2c3f747cd821eefd4e42d9880 (127.0.97.1:45691): tablet 59455a9b73f4403e9f6d718957336a53 (table test-table [id=5019798c2dac40a4ac7942def465aae3]) successfully deleted
I20260928 08:40:56.037137 16417 ts_tablet_manager.cc:1918] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:40:56.037312 16417 log.cc:1196] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/59455a9b73f4403e9f6d718957336a53
I20260928 08:40:56.037822 16417 ts_tablet_manager.cc:1939] T 59455a9b73f4403e9f6d718957336a53 P 91a5b07dea91468881c22fdeb4281e05: Deleting consensus metadata
Sep 28 08:40:56 dist-test-slave-w6c0 krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:40:56.039714 15902 catalog_manager.cc:4841] TS 91a5b07dea91468881c22fdeb4281e05 (127.0.97.2:36155): tablet 59455a9b73f4403e9f6d718957336a53 (table test-table [id=5019798c2dac40a4ac7942def465aae3]) successfully deleted
I20260928 08:40:56.045583 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:56.007327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35018 (local address 127.0.97.1:45691)
0928 08:40:56.007530 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:56.007540 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:56.027188 (+ 19648us) server_negotiation.cc:407] Connection header received
0928 08:40:56.027404 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:56.027410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:56.027500 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:56.027628 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:56.030034 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:56.031265 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:56.033854 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:56.034552 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:56.034650 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:56.039375 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:56.039386 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:56.039401 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:56.039475 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:56.042780 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:56.043471 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:56.043480 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:56.043484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:56.043640 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:56.044021 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:56.044026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:56.044029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:56.044360 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:56.044664 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:56.044968 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 08:40:56.045213 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:40:56.046267 16100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:56.146068 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 15977
I20260928 08:40:56.171377 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 16111
Sep 28 08:40:56 dist-test-slave-w6c0 krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:40:56.186640 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:56.172183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.0.97.2:36155)
0928 08:40:56.172354 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:56.172360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:56.172434 (+ 74us) server_negotiation.cc:407] Connection header received
0928 08:40:56.172662 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:56.172668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:56.172749 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:56.172858 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:56.174060 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:56.174961 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:56.176275 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:56.176668 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:56.176814 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:56.181045 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:56.181061 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:56.181078 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:56.181114 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:56.183808 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:56.184538 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:56.184545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:56.184547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:56.184666 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:56.185038 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:56.185043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:56.185048 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:56.185421 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:56.185745 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:56.186026 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 08:40:56.186238 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:40:56.187287 16234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:56.275767 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 16111
I20260928 08:40:56.295403 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 16244
Sep 28 08:40:56 dist-test-slave-w6c0 krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:40:56.310791 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:40:56.296252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.0.97.3:45131)
0928 08:40:56.296452 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 08:40:56.296456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:40:56.296475 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:40:56.296532 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:40:56.296536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:40:56.296604 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:40:56.296694 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:40:56.298090 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:56.298957 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:56.300410 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:40:56.300828 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:40:56.300930 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:40:56.305389 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:40:56.305404 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:40:56.305420 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:40:56.305452 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:40:56.308212 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:56.308862 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:56.308870 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:56.308873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:56.309008 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:40:56.309354 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:40:56.309357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:40:56.309359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:40:56.309764 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:40:56.309981 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 08:40:56.310282 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 08:40:56.310486 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 08:40:56.311471 16367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:56.401646 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 16244
I20260928 08:40:56.418723 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 15884
I20260928 08:40:56.419886 15946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:40:56.531637 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 15884
I20260928 08:41:16.553610 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34279
--webserver_interface=127.0.97.62
--webserver_port=43233
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:41:16.886181 16422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:41:16.886653 16422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:41:16.886832 16422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:41:16.894079 16422 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 08:41:16.894169 16422 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 08:41:16.894243 16422 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 08:41:16.894320 16422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:41:16.894420 16422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:41:16.894559 16422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:41:16.894649 16422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:41:16.894734 16422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:41:16.904820 16422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:41:16.906123 16422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:41:16.907490 16422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:41:16.930893 16428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:16.930953 16431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:16.930873 16429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:41:16.930894 16422 server_base.cc:1034] running on GCE node
I20260928 08:41:16.932209 16422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:41:16.933717 16422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:41:16.934942 16422 hybrid_clock.cc:648] HybridClock initialized: now 1790584876934874 us; error 39 us; skew 500 ppm
I20260928 08:41:16.935314 16422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:41:16 dist-test-slave-w6c0 krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:41:16.939131 16422 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:41:16.940699 16422 webserver.cc:460] Webserver started at http://127.0.97.62:43233/ using document root <none> and password file <none>
I20260928 08:41:16.941485 16422 fs_manager.cc:359] Metadata directory not provided
I20260928 08:41:16.941599 16422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:41:16.947754 16422 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 08:41:16.950659 16438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:16.951962 16422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:41:16.952157 16422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "cc0e87f714db45809e15f16e4f885f6a"
format_stamp: "Formatted at 2026-09-28 08:40:54 on dist-test-slave-w6c0"
server_key: "92deef5e48a24d9199249df5e307860f"
server_key_iv: "7d840456cef3b2551634f251c35fa6c2"
server_key_version: "encryptionkey@0"
I20260928 08:41:16.953203 16422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:41:16.969863 16422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:41:16.974046 16422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:41:16.974380 16422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:41:16.994210 16422 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34279
I20260928 08:41:16.994233 16489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34279 every 8 connection(s)
I20260928 08:41:16.995836 16422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:41:16.998356 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 16422
I20260928 08:41:16.999490 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45691
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34279
--webserver_port=33749
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:41:17.004628 16490 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:41:17.008028 16490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a: Bootstrap starting.
I20260928 08:41:17.028539 16490 log.cc:824] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:41:17.040402 16490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:41:17.040982 16490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a: Bootstrap complete.
I20260928 08:41:17.050588 16490 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0e87f714db45809e15f16e4f885f6a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34279 } }
I20260928 08:41:17.051470 16490 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc0e87f714db45809e15f16e4f885f6a, State: Initialized, Role: FOLLOWER
I20260928 08:41:17.052201 16490 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0e87f714db45809e15f16e4f885f6a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34279 } }
I20260928 08:41:17.052441 16490 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:41:17.052572 16490 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:41:17.052763 16490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:41:17.056663 16490 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0e87f714db45809e15f16e4f885f6a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34279 } }
I20260928 08:41:17.057251 16490 leader_election.cc:302] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc0e87f714db45809e15f16e4f885f6a; no voters:
I20260928 08:41:17.057933 16490 leader_election.cc:288] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 08:41:17.058120 16494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:41:17.059861 16490 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:41:17.065227 16494 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 2 LEADER]: Becoming Leader. State: Replica: cc0e87f714db45809e15f16e4f885f6a, State: Running, Role: LEADER
I20260928 08:41:17.065752 16494 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0e87f714db45809e15f16e4f885f6a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34279 } }
I20260928 08:41:17.068650 16495 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cc0e87f714db45809e15f16e4f885f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0e87f714db45809e15f16e4f885f6a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34279 } } }
I20260928 08:41:17.069079 16495 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [sys.catalog]: This master's current role is: LEADER
I20260928 08:41:17.070152 16509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:41:17.069980 16496 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc0e87f714db45809e15f16e4f885f6a. Latest consensus state: current_term: 2 leader_uuid: "cc0e87f714db45809e15f16e4f885f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0e87f714db45809e15f16e4f885f6a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34279 } } }
I20260928 08:41:17.070330 16496 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [sys.catalog]: This master's current role is: LEADER
I20260928 08:41:17.078020 16509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:41:17.079402 16509 catalog_manager.cc:1259] Loaded cluster ID: 33aa991d51c843ec93897e69432e97ea
I20260928 08:41:17.079510 16509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:41:17.083523 16509 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:41:17.084995 16509 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a: Loaded TSK: 0
I20260928 08:41:17.120065 16509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a: Generated new TSK 1
I20260928 08:41:17.120498 16509 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 08:41:17.340981 16492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:41:17.341346 16492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:41:17.341517 16492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:41:17.349004 16492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:41:17.349161 16492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:41:17.349336 16492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:41:17.359669 16492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=33749
--tserver_master_addrs=127.0.97.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:41:17.361174 16492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:41:17.362535 16492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:41:17.377341 16518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:17.377555 16520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:17.378189 16517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:41:17.379261 16492 server_base.cc:1034] running on GCE node
I20260928 08:41:17.379878 16492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:41:17.380898 16492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:41:17.382133 16492 hybrid_clock.cc:648] HybridClock initialized: now 1790584877382075 us; error 55 us; skew 500 ppm
I20260928 08:41:17.382474 16492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:41:17 dist-test-slave-w6c0 krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:41:17.392484 16492 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:41:17.399869 16492 webserver.cc:460] Webserver started at http://127.0.97.1:33749/ using document root <none> and password file <none>
I20260928 08:41:17.400648 16492 fs_manager.cc:359] Metadata directory not provided
I20260928 08:41:17.400802 16492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:41:17.406857 16492 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 08:41:17.409781 16527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:17.411209 16492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:41:17.411383 16492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "6a6185f2c3f747cd821eefd4e42d9880"
format_stamp: "Formatted at 2026-09-28 08:40:54 on dist-test-slave-w6c0"
server_key: "aa1d4bc936d8ffdef364e3eb5da6e187"
server_key_iv: "05b3147b670feb8bc4ab348d1cce1abf"
server_key_version: "encryptionkey@0"
I20260928 08:41:17.412346 16492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:41:17.460731 16492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:41:17.464989 16492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:41:17.465271 16492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:41:17.466334 16492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:41:17.467736 16492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:41:17.467821 16492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:17.467973 16492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:41:17.468078 16492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:17.501856 16492 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45691
I20260928 08:41:17.501967 16639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45691 every 8 connection(s)
I20260928 08:41:17.503448 16492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:41:17.507121 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 16492
I20260928 08:41:17.507833 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36155
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34279
--webserver_port=41923
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:41:17 dist-test-slave-w6c0 krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:41:17.534480 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:17.506235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52889 (local address 127.0.97.62:34279)
0928 08:41:17.506967 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:17.506985 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:17.507596 (+ 611us) server_negotiation.cc:407] Connection header received
0928 08:41:17.509928 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:17.509950 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:17.514875 (+ 4925us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:17.515285 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:41:17.517450 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:17.518775 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:17.521265 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:17.521906 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:17.522046 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:17.526695 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:41:17.526703 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:41:17.526719 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:41:17.526768 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:41:17.529763 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:17.530800 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:17.530807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:17.530812 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:17.530971 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:17.531561 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:17.531565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:17.531567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:17.532055 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:41:17.532199 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:17.533200 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0928 08:41:17.533523 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":336,"threads_started":1}
I20260928 08:41:17.535060 16643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:17.506459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:52889)
0928 08:41:17.507381 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:17.507436 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 08:41:17.509334 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:41:17.515923 (+ 6589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:41:17.515935 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:41:17.516505 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:41:17.517220 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:17.517248 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:17.518957 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:17.518966 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:17.521112 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:17.521130 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:17.522166 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:17.522175 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:17.522526 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:17.523273 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:41:17.523309 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:41:17.526326 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:41:17.529920 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:41:17.529931 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:41:17.529949 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:41:17.530666 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:41:17.531111 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:41:17.531119 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:41:17.531124 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:41:17.531391 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:41:17.532220 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:41:17.532232 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:41:17.532684 (+ 452us) client_negotiation.cc:769] Sending connection context
0928 08:41:17.533037 (+ 353us) client_negotiation.cc:240] Negotiation successful
0928 08:41:17.533641 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":339,"threads_started":1}
I20260928 08:41:17.537767 16640 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34279
I20260928 08:41:17.538117 16640 heartbeater.cc:464] Registering TS with master...
I20260928 08:41:17.538883 16640 heartbeater.cc:511] Master 127.0.97.62:34279 requested a full tablet report, sending...
I20260928 08:41:17.540958 16455 ts_manager.cc:194] Registered new tserver with Master: 6a6185f2c3f747cd821eefd4e42d9880 (127.0.97.1:45691)
I20260928 08:41:17.542448 16455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52889
W20260928 08:41:17.845363 16644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:41:17.845700 16644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:41:17.845880 16644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:41:17.852994 16644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:41:17.853148 16644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:41:17.853343 16644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:41:17.863797 16644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=41923
--tserver_master_addrs=127.0.97.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:41:17.865120 16644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:41:17.866437 16644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:41:17.879563 16649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:17.879509 16650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:17.879544 16652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:41:17.880611 16644 server_base.cc:1034] running on GCE node
I20260928 08:41:17.881196 16644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:41:17.882087 16644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:41:17.883311 16644 hybrid_clock.cc:648] HybridClock initialized: now 1790584877883278 us; error 48 us; skew 500 ppm
I20260928 08:41:17.883618 16644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:41:17 dist-test-slave-w6c0 krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:41:17.886790 16644 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:41:17.888286 16644 webserver.cc:460] Webserver started at http://127.0.97.2:41923/ using document root <none> and password file <none>
I20260928 08:41:17.889074 16644 fs_manager.cc:359] Metadata directory not provided
I20260928 08:41:17.889251 16644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:41:17.894577 16644 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:41:17.897092 16659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:17.898331 16644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:41:17.898586 16644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "91a5b07dea91468881c22fdeb4281e05"
format_stamp: "Formatted at 2026-09-28 08:40:55 on dist-test-slave-w6c0"
server_key: "8bb41703a271c4f986871ddcfd31db31"
server_key_iv: "aec12794edd659d5a20f44817875be23"
server_key_version: "encryptionkey@0"
I20260928 08:41:17.899529 16644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:41:17.912652 16644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:41:17.916929 16644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:41:17.917210 16644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:41:17.918133 16644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:41:17.919482 16644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:41:17.919611 16644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:17.919752 16644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:41:17.919844 16644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:17.953682 16644 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36155
I20260928 08:41:17.953768 16771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36155 every 8 connection(s)
I20260928 08:41:17.955291 16644 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:41:17.965729 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 16644
I20260928 08:41:17.966495 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45131
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34279
--webserver_port=38389
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:41:17 dist-test-slave-w6c0 krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:41:17.975850 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:17.958029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48469 (local address 127.0.97.62:34279)
0928 08:41:17.958216 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:17.958220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:17.959387 (+ 1167us) server_negotiation.cc:407] Connection header received
0928 08:41:17.961339 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:17.961345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:17.961456 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:17.961590 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:41:17.963206 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:17.963915 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:17.965468 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:17.965995 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:17.966109 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:17.970284 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:41:17.970290 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:41:17.970293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:41:17.970327 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:41:17.972875 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:17.973603 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:17.973607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:17.973609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:17.973728 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:17.974164 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:17.974175 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:17.974178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:17.974441 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:41:17.974624 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:17.975271 (+ 647us) server_negotiation.cc:299] Negotiation successful
0928 08:41:17.975504 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:41:17.976761 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:17.958428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:48469)
0928 08:41:17.959222 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:17.959265 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:41:17.961053 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:41:17.961818 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:41:17.961826 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:41:17.962399 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:41:17.963037 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:17.963053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:17.964083 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:17.964089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:17.965311 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:17.965335 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:17.966228 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:17.966236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:17.966570 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:17.967294 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:41:17.967329 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:41:17.970033 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:41:17.973019 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:41:17.973028 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:41:17.973043 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:41:17.973459 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:41:17.973842 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:41:17.973847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:41:17.973850 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:41:17.974036 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:41:17.974596 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:41:17.974603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:41:17.974912 (+ 309us) client_negotiation.cc:769] Sending connection context
0928 08:41:17.975172 (+ 260us) client_negotiation.cc:240] Negotiation successful
0928 08:41:17.975666 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":302,"threads_started":1}
I20260928 08:41:17.978263 16772 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34279
I20260928 08:41:17.978644 16772 heartbeater.cc:464] Registering TS with master...
I20260928 08:41:17.979360 16772 heartbeater.cc:511] Master 127.0.97.62:34279 requested a full tablet report, sending...
I20260928 08:41:17.980786 16455 ts_manager.cc:194] Registered new tserver with Master: 91a5b07dea91468881c22fdeb4281e05 (127.0.97.2:36155)
I20260928 08:41:17.981587 16455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48469
W20260928 08:41:18.298792 16775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:41:18.299129 16775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:41:18.299302 16775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:41:18.306717 16775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:41:18.306874 16775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:41:18.307055 16775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:41:18.317057 16775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38389
--tserver_master_addrs=127.0.97.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:41:18.318372 16775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:41:18.319682 16775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:41:18.333719 16780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:18.333773 16781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:18.333799 16783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:41:18.334347 16775 server_base.cc:1034] running on GCE node
I20260928 08:41:18.335004 16775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:41:18.335927 16775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:41:18.337137 16775 hybrid_clock.cc:648] HybridClock initialized: now 1790584878337105 us; error 33 us; skew 500 ppm
I20260928 08:41:18.337404 16775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:41:18 dist-test-slave-w6c0 krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:41:18.340488 16775 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:41:18.341897 16775 webserver.cc:460] Webserver started at http://127.0.97.3:38389/ using document root <none> and password file <none>
I20260928 08:41:18.342661 16775 fs_manager.cc:359] Metadata directory not provided
I20260928 08:41:18.342820 16775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:41:18.348023 16775 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:41:18.350721 16790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:18.352015 16775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:41:18.352198 16775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "e491b4648bcd40bab94d37e206ebaa6a"
format_stamp: "Formatted at 2026-09-28 08:40:55 on dist-test-slave-w6c0"
server_key: "441fe1d7e9880e188ac4f5c169142826"
server_key_iv: "44d7cf3fd5639f9125a6e2f9bb40136a"
server_key_version: "encryptionkey@0"
I20260928 08:41:18.353157 16775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:41:18.401382 16775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:41:18.406204 16775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:41:18.406582 16775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:41:18.407653 16775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:41:18.409129 16775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:41:18.409243 16775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:18.409375 16775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:41:18.409476 16775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:18.444569 16775 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45131
I20260928 08:41:18.444662 16902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45131 every 8 connection(s)
I20260928 08:41:18.446089 16775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:41:18.452399 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 16775
Sep 28 08:41:18 dist-test-slave-w6c0 krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:41:18.470037 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.448838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35123 (local address 127.0.97.62:34279)
0928 08:41:18.449039 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:18.449043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:18.450145 (+ 1102us) server_negotiation.cc:407] Connection header received
0928 08:41:18.452067 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:18.452078 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:18.452196 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:18.452313 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:41:18.454128 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.455072 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.457331 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.457899 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.458031 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:18.462234 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:41:18.462239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:41:18.462242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:41:18.462274 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:41:18.466263 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:18.467188 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:18.467195 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:18.467201 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:18.467345 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:18.467852 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:18.467855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:18.467857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:18.468265 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:41:18.468368 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:18.469315 (+ 947us) server_negotiation.cc:299] Negotiation successful
0928 08:41:18.469622 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":71,"spinlock_wait_cycles":768}
I20260928 08:41:18.470355 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.453309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42074 (local address 127.0.97.62:34279)
0928 08:41:18.454176 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:18.454183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:18.454205 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:41:18.454266 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:18.454270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:18.454338 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:18.454459 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:41:18.456498 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.457479 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.459465 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.460126 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.460261 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:18.463531 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:41:18.463537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:41:18.463542 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:41:18.463595 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:41:18.467143 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:18.467969 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:18.467974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:18.467977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:18.468109 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:18.468501 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:18.468504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:18.468506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:18.468744 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:41:18.468940 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:18.469315 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 08:41:18.469924 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":202,"server-negotiator.queue_time_us":756,"spinlock_wait_cycles":5632,"thread_start_us":639,"threads_started":1}
I20260928 08:41:18.471160 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.449252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:35123)
0928 08:41:18.449978 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:18.450017 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:41:18.451776 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:41:18.452582 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:41:18.452596 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:41:18.453138 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:41:18.453883 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:18.453909 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.455257 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.455265 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:18.457204 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:18.457216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.458140 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.458149 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:18.458416 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:18.459265 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:41:18.459288 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:41:18.462006 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:41:18.466420 (+ 4414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:41:18.466427 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:41:18.466443 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:41:18.467075 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:41:18.467458 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:41:18.467463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:41:18.467466 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:41:18.467725 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:41:18.468389 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:41:18.468398 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:41:18.468814 (+ 416us) client_negotiation.cc:769] Sending connection context
0928 08:41:18.469165 (+ 351us) client_negotiation.cc:240] Negotiation successful
0928 08:41:18.469710 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":303,"threads_started":1}
I20260928 08:41:18.472757 16903 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34279
I20260928 08:41:18.473152 16903 heartbeater.cc:464] Registering TS with master...
I20260928 08:41:18.473877 16903 heartbeater.cc:511] Master 127.0.97.62:34279 requested a full tablet report, sending...
I20260928 08:41:18.475322 16455 ts_manager.cc:194] Registered new tserver with Master: e491b4648bcd40bab94d37e206ebaa6a (127.0.97.3:45131)
I20260928 08:41:18.476117 16455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35123
I20260928 08:41:18.485430 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:41:18.492429 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.486586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.0.97.62:34279)
0928 08:41:18.486833 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:18.486839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:18.487101 (+ 262us) server_negotiation.cc:407] Connection header received
0928 08:41:18.487354 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:18.487357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:18.487503 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:18.487665 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:41:18.488113 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.488839 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.490312 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.490863 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.491016 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:18.491511 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:41:18.491735 (+ 224us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 08:41:18.492149 (+ 414us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42076: token expired
Metrics: {"server-negotiator.queue_time_us":91}
W20260928 08:41:18.492589 16906 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42076: token expired
W20260928 08:41:18.493067 16378 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:34279)
I20260928 08:41:18.506594 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.493603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.0.97.62:34279)
0928 08:41:18.493760 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:18.493764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:18.493780 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:41:18.493901 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:18.493904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:18.493962 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:18.494049 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:41:18.495457 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.496181 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.497428 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.497819 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.497905 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:18.500556 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:41:18.500561 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:41:18.500564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:41:18.500594 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:41:18.503863 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:18.504620 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:18.504625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:18.504629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:18.504792 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:18.505135 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:18.505141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:18.505145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:18.505542 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:41:18.505801 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:18.506023 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 08:41:18.506238 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:41:18.510218 16455 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:41:18.517527 16455 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:41:18.539369 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.529354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.0.97.2:36155)
0928 08:41:18.531653 (+ 2299us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:18.531666 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:18.531699 (+ 33us) server_negotiation.cc:407] Connection header received
0928 08:41:18.531803 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:18.531811 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:18.532074 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:18.533120 (+ 1046us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:41:18.533349 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.534446 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.537129 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.538137 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.538271 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:18.538464 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:18.538851 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 08:41:18.539026 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2137,"thread_start_us":2040,"threads_started":1}
I20260928 08:41:18.539425 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.529153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36155 (local address 127.0.0.1:52748)
0928 08:41:18.530097 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:18.530136 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:41:18.530303 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:41:18.532303 (+ 2000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:41:18.532310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:41:18.532337 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:41:18.532875 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:18.532888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.534692 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.534697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:18.536983 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:18.536995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.538365 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.538371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:18.538616 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:18.538670 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 08:41:18.538802 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 08:41:18.538996 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":2,"thread_start_us":1492,"threads_started":1}
I20260928 08:41:18.540822 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.529296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45131 (local address 127.0.0.1:42054)
0928 08:41:18.531006 (+ 1710us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:18.531029 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:41:18.531159 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:41:18.532896 (+ 1737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:41:18.532900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:41:18.532939 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:41:18.533347 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:18.533359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.535480 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.535485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:18.537908 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:18.537924 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.539999 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.540006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:18.540197 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:18.540231 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 08:41:18.540320 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 08:41:18.540455 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1538,"thread_start_us":429,"threads_started":1}
I20260928 08:41:18.541534 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.530066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.0.97.3:45131)
0928 08:41:18.532032 (+ 1966us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:18.532040 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:18.532071 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:41:18.532173 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:18.532182 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:18.532384 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:18.532586 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:41:18.533545 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.535322 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.538084 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.539500 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.539628 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:18.539827 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:18.540862 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0928 08:41:18.541236 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1809,"thread_start_us":1662,"threads_started":1}
I20260928 08:41:18.541559 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.531713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.0.97.1:45691)
0928 08:41:18.532439 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:18.532450 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:18.533060 (+ 610us) server_negotiation.cc:407] Connection header received
0928 08:41:18.533374 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:18.533381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:18.533591 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:18.533750 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:41:18.534600 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.535739 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.538621 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.540107 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.540234 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:18.540451 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:18.541037 (+ 586us) server_negotiation.cc:299] Negotiation successful
0928 08:41:18.541203 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":378,"threads_started":1}
I20260928 08:41:18.542173 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.531149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45691 (local address 127.0.0.1:41730)
0928 08:41:18.532942 (+ 1793us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:18.532960 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:41:18.533171 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:41:18.534047 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:41:18.534052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:41:18.534076 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:41:18.534412 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:18.534422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.535900 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.535905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:18.538398 (+ 2493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:18.538410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.540632 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.540639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:18.540819 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:18.540859 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 08:41:18.541762 (+ 903us) client_negotiation.cc:240] Negotiation successful
0928 08:41:18.541898 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1670,"mutex_wait_us":4,"thread_start_us":1503,"threads_started":1}
I20260928 08:41:18.544126 16707 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5341713efc434b86c821b6ca69ecca (DEFAULT_TABLE table=test-table [id=035076300ae2479ab2bccd7017d3fe54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:41:18.545507 16640 heartbeater.cc:503] Master 127.0.97.62:34279 was elected leader, sending a full tablet report...
I20260928 08:41:18.545678 16707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5341713efc434b86c821b6ca69ecca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:41:18.545624 16575 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5341713efc434b86c821b6ca69ecca (DEFAULT_TABLE table=test-table [id=035076300ae2479ab2bccd7017d3fe54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:41:18.545603 16838 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5341713efc434b86c821b6ca69ecca (DEFAULT_TABLE table=test-table [id=035076300ae2479ab2bccd7017d3fe54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:41:18.546980 16575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5341713efc434b86c821b6ca69ecca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:41:18.546986 16838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5341713efc434b86c821b6ca69ecca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:41:18.560170 16919 tablet_bootstrap.cc:492] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a: Bootstrap starting.
I20260928 08:41:18.564137 16919 tablet_bootstrap.cc:654] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a: Neither blocks nor log segments found. Creating new log.
I20260928 08:41:18.565223 16919 log.cc:824] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:41:18.565299 16921 tablet_bootstrap.cc:492] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880: Bootstrap starting.
I20260928 08:41:18.566480 16920 tablet_bootstrap.cc:492] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05: Bootstrap starting.
I20260928 08:41:18.569059 16921 tablet_bootstrap.cc:654] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880: Neither blocks nor log segments found. Creating new log.
I20260928 08:41:18.570196 16920 tablet_bootstrap.cc:654] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05: Neither blocks nor log segments found. Creating new log.
I20260928 08:41:18.570340 16921 log.cc:824] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880: Log is configured to *not* fsync() on all Append() calls
I20260928 08:41:18.571288 16919 tablet_bootstrap.cc:492] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a: No bootstrap required, opened a new log
I20260928 08:41:18.571566 16919 ts_tablet_manager.cc:1397] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260928 08:41:18.571547 16920 log.cc:824] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05: Log is configured to *not* fsync() on all Append() calls
I20260928 08:41:18.575119 16920 tablet_bootstrap.cc:492] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05: No bootstrap required, opened a new log
I20260928 08:41:18.575438 16920 ts_tablet_manager.cc:1397] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260928 08:41:18.579797 16921 tablet_bootstrap.cc:492] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880: No bootstrap required, opened a new log
I20260928 08:41:18.579706 16919 raft_consensus.cc:348] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } }
I20260928 08:41:18.580076 16921 ts_tablet_manager.cc:1397] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260928 08:41:18.580128 16919 raft_consensus.cc:374] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:41:18.580230 16919 raft_consensus.cc:729] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e491b4648bcd40bab94d37e206ebaa6a, State: Initialized, Role: FOLLOWER
I20260928 08:41:18.580832 16919 consensus_queue.cc:260] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } }
I20260928 08:41:18.586198 16920 raft_consensus.cc:348] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } }
I20260928 08:41:18.586709 16920 raft_consensus.cc:374] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:41:18.586815 16920 raft_consensus.cc:729] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91a5b07dea91468881c22fdeb4281e05, State: Initialized, Role: FOLLOWER
I20260928 08:41:18.587473 16903 heartbeater.cc:503] Master 127.0.97.62:34279 was elected leader, sending a full tablet report...
I20260928 08:41:18.587527 16920 consensus_queue.cc:260] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } }
I20260928 08:41:18.588809 16919 ts_tablet_manager.cc:1428] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
I20260928 08:41:18.589776 16772 heartbeater.cc:503] Master 127.0.97.62:34279 was elected leader, sending a full tablet report...
I20260928 08:41:18.590382 16920 ts_tablet_manager.cc:1428] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05: Time spent starting tablet: real 0.015s user 0.013s sys 0.002s
I20260928 08:41:18.592120 16921 raft_consensus.cc:348] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } }
I20260928 08:41:18.592597 16921 raft_consensus.cc:374] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:41:18.592722 16921 raft_consensus.cc:729] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6185f2c3f747cd821eefd4e42d9880, State: Initialized, Role: FOLLOWER
I20260928 08:41:18.593317 16921 consensus_queue.cc:260] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } }
I20260928 08:41:18.595662 16921 ts_tablet_manager.cc:1428] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260928 08:41:18.598369 16926 raft_consensus.cc:482] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:41:18.598650 16926 raft_consensus.cc:504] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } }
I20260928 08:41:18.600185 16926 leader_election.cc:288] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e491b4648bcd40bab94d37e206ebaa6a (127.0.97.3:45131), 6a6185f2c3f747cd821eefd4e42d9880 (127.0.97.1:45691)
I20260928 08:41:18.608785 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.600543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39867 (local address 127.0.97.1:45691)
0928 08:41:18.600772 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:18.600781 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:18.603498 (+ 2717us) server_negotiation.cc:407] Connection header received
0928 08:41:18.603684 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:18.603688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:18.603741 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:18.603841 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:41:18.604420 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.605108 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.606970 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.607757 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.607922 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:18.608032 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:18.608394 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 08:41:18.608537 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:41:18.608783 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.600440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56469 (local address 127.0.97.3:45131)
0928 08:41:18.600646 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:18.600654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:18.603178 (+ 2524us) server_negotiation.cc:407] Connection header received
0928 08:41:18.603404 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:18.603409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:18.603491 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:18.603587 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:41:18.604157 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.604972 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.606935 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.607710 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.607868 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:18.607985 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:18.608325 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 08:41:18.608514 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:41:18.609025 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.600329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45131 (local address 127.0.97.2:56469)
0928 08:41:18.603063 (+ 2734us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:18.603086 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:41:18.603268 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:41:18.603600 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:41:18.603605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:41:18.603630 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:41:18.603999 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:18.604011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.605113 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.605119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:18.606783 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:18.606795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.607950 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.607957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:18.608133 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:18.608192 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 08:41:18.608299 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 08:41:18.608454 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2625,"mutex_wait_us":93,"thread_start_us":2416,"threads_started":1}
I20260928 08:41:18.609078 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.600528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45691 (local address 127.0.97.2:39867)
0928 08:41:18.603416 (+ 2888us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:18.603431 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:41:18.603539 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:41:18.603858 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:41:18.603862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:41:18.603886 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:41:18.604249 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:18.604256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.605244 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.605250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:18.606820 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:18.606832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.607997 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.608003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:18.608228 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:18.608252 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 08:41:18.608354 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 08:41:18.608506 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2766,"mutex_wait_us":61,"thread_start_us":2568,"threads_started":1}
I20260928 08:41:18.610018 16595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5341713efc434b86c821b6ca69ecca" candidate_uuid: "91a5b07dea91468881c22fdeb4281e05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6185f2c3f747cd821eefd4e42d9880" is_pre_election: true
I20260928 08:41:18.610071 16858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5341713efc434b86c821b6ca69ecca" candidate_uuid: "91a5b07dea91468881c22fdeb4281e05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e491b4648bcd40bab94d37e206ebaa6a" is_pre_election: true
I20260928 08:41:18.610622 16858 raft_consensus.cc:2454] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91a5b07dea91468881c22fdeb4281e05 in term 0.
I20260928 08:41:18.610606 16595 raft_consensus.cc:2454] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91a5b07dea91468881c22fdeb4281e05 in term 0.
I20260928 08:41:18.611297 16662 leader_election.cc:302] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a6185f2c3f747cd821eefd4e42d9880, 91a5b07dea91468881c22fdeb4281e05; no voters:
I20260928 08:41:18.611740 16926 raft_consensus.cc:2781] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:41:18.611905 16926 raft_consensus.cc:482] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:41:18.612016 16926 raft_consensus.cc:3037] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:41:18.614271 16926 raft_consensus.cc:504] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } }
I20260928 08:41:18.614892 16926 leader_election.cc:288] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [CANDIDATE]: Term 1 election: Requested vote from peers e491b4648bcd40bab94d37e206ebaa6a (127.0.97.3:45131), 6a6185f2c3f747cd821eefd4e42d9880 (127.0.97.1:45691)
I20260928 08:41:18.615361 16858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5341713efc434b86c821b6ca69ecca" candidate_uuid: "91a5b07dea91468881c22fdeb4281e05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e491b4648bcd40bab94d37e206ebaa6a"
I20260928 08:41:18.615374 16595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5341713efc434b86c821b6ca69ecca" candidate_uuid: "91a5b07dea91468881c22fdeb4281e05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6185f2c3f747cd821eefd4e42d9880"
I20260928 08:41:18.615571 16858 raft_consensus.cc:3037] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:41:18.615644 16595 raft_consensus.cc:3037] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:41:18.617928 16858 raft_consensus.cc:2454] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91a5b07dea91468881c22fdeb4281e05 in term 1.
I20260928 08:41:18.618033 16595 raft_consensus.cc:2454] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91a5b07dea91468881c22fdeb4281e05 in term 1.
I20260928 08:41:18.618355 16663 leader_election.cc:302] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91a5b07dea91468881c22fdeb4281e05, e491b4648bcd40bab94d37e206ebaa6a; no voters:
I20260928 08:41:18.618744 16926 raft_consensus.cc:2781] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:41:18.619225 16926 raft_consensus.cc:686] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [term 1 LEADER]: Becoming Leader. State: Replica: 91a5b07dea91468881c22fdeb4281e05, State: Running, Role: LEADER
I20260928 08:41:18.619791 16926 consensus_queue.cc:237] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } }
I20260928 08:41:18.624872 16454 catalog_manager.cc:5495] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91a5b07dea91468881c22fdeb4281e05 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "91a5b07dea91468881c22fdeb4281e05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } health_report { overall_health: UNKNOWN } } }
I20260928 08:41:18.643603 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.638299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.0.97.2:36155)
0928 08:41:18.638551 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:18.638559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:18.638586 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:41:18.638689 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:18.638693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:18.638766 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:18.638918 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:41:18.639276 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.640006 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.641349 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.641754 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.641890 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:18.642383 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:41:18.642973 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:41:18.643140 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:18.643243 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 08:41:18.643328 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:41:18.652164 16595 raft_consensus.cc:1261] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Refusing update from remote peer 91a5b07dea91468881c22fdeb4281e05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:41:18.652165 16858 raft_consensus.cc:1261] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a [term 1 FOLLOWER]: Refusing update from remote peer 91a5b07dea91468881c22fdeb4281e05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:41:18.653174 16930 consensus_queue.cc:1035] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:41:18.653514 16926 consensus_queue.cc:1035] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:41:18.681952 16455 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:41:18.682271 16455 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:41:18.685935 16455 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a: Sending DeleteTablet for 3 replicas of tablet 1a5341713efc434b86c821b6ca69ecca
I20260928 08:41:18.687201 16838 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a5341713efc434b86c821b6ca69ecca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:41:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42054
I20260928 08:41:18.687206 16707 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a5341713efc434b86c821b6ca69ecca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:41:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52748
I20260928 08:41:18.688160 16942 tablet_replica.cc:330] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a: stopping tablet replica
I20260928 08:41:18.688663 16942 raft_consensus.cc:2229] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:41:18.688766 16575 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a5341713efc434b86c821b6ca69ecca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:41:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41730
I20260928 08:41:18.689390 16942 raft_consensus.cc:2258] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:41:18.690629 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 16492
I20260928 08:41:18.692355 16942 ts_tablet_manager.cc:1905] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:41:18.692963 16943 tablet_replica.cc:330] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05: stopping tablet replica
I20260928 08:41:18.693451 16943 raft_consensus.cc:2229] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:41:18.694154 16943 raft_consensus.cc:2258] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:41:18.698848 16944 tablet_replica.cc:330] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880: stopping tablet replica
I20260928 08:41:18.700045 16943 ts_tablet_manager.cc:1905] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:41:18.702235 16942 ts_tablet_manager.cc:1918] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:41:18.702414 16942 log.cc:1196] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/1a5341713efc434b86c821b6ca69ecca
I20260928 08:41:18.703133 16942 ts_tablet_manager.cc:1939] T 1a5341713efc434b86c821b6ca69ecca P e491b4648bcd40bab94d37e206ebaa6a: Deleting consensus metadata
I20260928 08:41:18.705328 16442 catalog_manager.cc:4841] TS e491b4648bcd40bab94d37e206ebaa6a (127.0.97.3:45131): tablet 1a5341713efc434b86c821b6ca69ecca (table test-table [id=035076300ae2479ab2bccd7017d3fe54]) successfully deleted
I20260928 08:41:18.703023 16944 raft_consensus.cc:2229] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:41:18.707909 16944 raft_consensus.cc:2258] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:41:18.708835 16943 ts_tablet_manager.cc:1918] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:41:18.709074 16943 log.cc:1196] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/1a5341713efc434b86c821b6ca69ecca
I20260928 08:41:18.709697 16943 ts_tablet_manager.cc:1939] T 1a5341713efc434b86c821b6ca69ecca P 91a5b07dea91468881c22fdeb4281e05: Deleting consensus metadata
I20260928 08:41:18.709852 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.692028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41742 (local address 127.0.97.1:45691)
0928 08:41:18.692241 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:18.692250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:18.692277 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:41:18.692354 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:18.692363 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:18.692481 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:18.692616 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:41:18.694495 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.696385 (+ 1890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.698308 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.699083 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.699222 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:18.702844 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:41:18.702861 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:41:18.702880 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:41:18.702928 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:41:18.706699 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:18.707570 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:18.707583 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:18.707586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:18.707712 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:18.708143 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:18.708148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:18.708153 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:18.708537 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:41:18.708895 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:18.709176 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 08:41:18.709413 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:41:18.710686 16635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:41:18.711851 16441 catalog_manager.cc:4841] TS 91a5b07dea91468881c22fdeb4281e05 (127.0.97.2:36155): tablet 1a5341713efc434b86c821b6ca69ecca (table test-table [id=035076300ae2479ab2bccd7017d3fe54]) successfully deleted
I20260928 08:41:18.804749 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 16492
I20260928 08:41:18.823312 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 16644
W20260928 08:41:18.823504 16441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45691 (error 108)
I20260928 08:41:18.826303 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.825781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45691 (local address 127.0.0.1:41756)
0928 08:41:18.825970 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:18.826103 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 08:41:18.826606 16441 catalog_manager.cc:4584] TS 6a6185f2c3f747cd821eefd4e42d9880 (127.0.97.1:45691): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a5341713efc434b86c821b6ca69ecca: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45691: connect: Connection refused (error 111)
I20260928 08:41:18.838709 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.824213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52774 (local address 127.0.97.2:36155)
0928 08:41:18.824418 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:18.824426 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:18.824671 (+ 245us) server_negotiation.cc:407] Connection header received
0928 08:41:18.824859 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:18.824866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:18.824973 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:18.825088 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:41:18.826943 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.827816 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.829156 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.829613 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.829753 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:18.832500 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:41:18.832519 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:41:18.832541 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:41:18.832591 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:41:18.835766 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:18.836468 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:18.836480 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:18.836486 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:18.836630 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:18.837027 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:18.837033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:18.837036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:18.837408 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:41:18.837763 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:18.838080 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 08:41:18.838314 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 08:41:18.839411 16767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:41:18.880571 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.880026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45691 (local address 127.0.0.1:41766)
0928 08:41:18.880260 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:18.880372 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 08:41:18.881635 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.881288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45691 (local address 127.0.0.1:41774)
0928 08:41:18.881415 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:18.881507 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 08:41:18.942322 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.941748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45691 (local address 127.0.0.1:41788)
0928 08:41:18.942051 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:18.942148 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260928 08:41:18.943521 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.943159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45691 (local address 127.0.0.1:41792)
0928 08:41:18.943316 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:18.943397 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 08:41:18.954658 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 16644
I20260928 08:41:18.974439 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 16775
I20260928 08:41:18.987553 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:18.975263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42070 (local address 127.0.97.3:45131)
0928 08:41:18.975430 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:18.975435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:18.975453 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:41:18.975607 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:18.975611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:18.975681 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:18.975769 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:41:18.977025 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.977748 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.979087 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:18.979470 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:18.979565 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:18.982181 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:41:18.982206 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:41:18.982223 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:41:18.982254 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:41:18.984953 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:18.985559 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:18.985565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:18.985567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:18.985690 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:18.985991 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:18.985994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:18.985996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:18.986342 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:41:18.986641 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:18.987106 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 08:41:18.987272 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 08:41:18.988216 16898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:41:19.024786 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:19.024299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45691 (local address 127.0.0.1:41794)
0928 08:41:19.024516 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:19.024628 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 08:41:19.025892 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:19.025504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45691 (local address 127.0.0.1:41810)
0928 08:41:19.025716 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:19.025783 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 08:41:19.082130 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 16775
I20260928 08:41:19.100240 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 16422
I20260928 08:41:19.101415 16485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:41:19.221500 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:19.220821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45691 (local address 127.0.0.1:41818)
0928 08:41:19.221062 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:19.221204 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 08:41:19.221738 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 16422
I20260928 08:41:39.244796 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34279
--webserver_interface=127.0.97.62
--webserver_port=43233
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:41:39.592090 16950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:41:39.592442 16950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:41:39.592595 16950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:41:39.601204 16950 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 08:41:39.601327 16950 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 08:41:39.601367 16950 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 08:41:39.601455 16950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:41:39.601573 16950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:41:39.601676 16950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:41:39.601795 16950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:41:39.601909 16950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:41:39.613174 16950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:41:39.614717 16950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:41:39.616165 16950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:41:39.630604 16958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:39.630543 16956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:39.630621 16955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:41:39.636370 16950 server_base.cc:1034] running on GCE node
I20260928 08:41:39.637414 16950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:41:39.639129 16950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:41:39.640406 16950 hybrid_clock.cc:648] HybridClock initialized: now 1790584899640345 us; error 44 us; skew 500 ppm
I20260928 08:41:39.640792 16950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:41:39 dist-test-slave-w6c0 krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:41:39.650758 16950 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:41:39.653153 16950 webserver.cc:460] Webserver started at http://127.0.97.62:43233/ using document root <none> and password file <none>
I20260928 08:41:39.654271 16950 fs_manager.cc:359] Metadata directory not provided
I20260928 08:41:39.654426 16950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:41:39.662164 16950 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:41:39.667268 16965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:39.669226 16950 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260928 08:41:39.669564 16950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "cc0e87f714db45809e15f16e4f885f6a"
format_stamp: "Formatted at 2026-09-28 08:40:54 on dist-test-slave-w6c0"
server_key: "92deef5e48a24d9199249df5e307860f"
server_key_iv: "7d840456cef3b2551634f251c35fa6c2"
server_key_version: "encryptionkey@0"
I20260928 08:41:39.670830 16950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:41:39.689565 16950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:41:39.694289 16950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:41:39.694654 16950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:41:39.714965 16950 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34279
I20260928 08:41:39.715166 17016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34279 every 8 connection(s)
I20260928 08:41:39.716564 16950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:41:39.721992 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 16950
I20260928 08:41:39.723502 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45691
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34279
--webserver_port=33749
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:41:39.724303 17017 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:41:39.727861 17017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a: Bootstrap starting.
I20260928 08:41:39.758800 17017 log.cc:824] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:41:39.782655 17017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:41:39.783397 17017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a: Bootstrap complete.
I20260928 08:41:39.798629 17017 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0e87f714db45809e15f16e4f885f6a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34279 } }
I20260928 08:41:39.800206 17017 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc0e87f714db45809e15f16e4f885f6a, State: Initialized, Role: FOLLOWER
I20260928 08:41:39.801051 17017 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0e87f714db45809e15f16e4f885f6a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34279 } }
I20260928 08:41:39.801397 17017 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:41:39.801540 17017 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:41:39.801728 17017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 2 FOLLOWER]: Advancing to term 3
I20260928 08:41:39.807657 17017 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0e87f714db45809e15f16e4f885f6a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34279 } }
I20260928 08:41:39.808319 17017 leader_election.cc:302] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc0e87f714db45809e15f16e4f885f6a; no voters:
I20260928 08:41:39.809085 17017 leader_election.cc:288] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 08:41:39.809289 17021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 3 FOLLOWER]: Leader election won for term 3
I20260928 08:41:39.810578 17021 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [term 3 LEADER]: Becoming Leader. State: Replica: cc0e87f714db45809e15f16e4f885f6a, State: Running, Role: LEADER
I20260928 08:41:39.811192 17021 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0e87f714db45809e15f16e4f885f6a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34279 } }
I20260928 08:41:39.811318 17017 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:41:39.814935 17022 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cc0e87f714db45809e15f16e4f885f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0e87f714db45809e15f16e4f885f6a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34279 } } }
I20260928 08:41:39.814913 17023 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc0e87f714db45809e15f16e4f885f6a. Latest consensus state: current_term: 3 leader_uuid: "cc0e87f714db45809e15f16e4f885f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0e87f714db45809e15f16e4f885f6a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34279 } } }
I20260928 08:41:39.815465 17022 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [sys.catalog]: This master's current role is: LEADER
I20260928 08:41:39.815512 17023 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a [sys.catalog]: This master's current role is: LEADER
I20260928 08:41:39.816764 17027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:41:39.839931 17027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:41:39.842208 17027 catalog_manager.cc:1259] Loaded cluster ID: 33aa991d51c843ec93897e69432e97ea
I20260928 08:41:39.842377 17027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:41:39.848150 17027 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:41:39.850611 17027 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a: Loaded TSK: 0
I20260928 08:41:39.850808 17027 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cc0e87f714db45809e15f16e4f885f6a: Loaded TSK: 1
F20260928 08:41:39.852327 17027 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff2e49fc9ad google::LogMessage::Fail() at ??:0
@ 0x7ff2e49ff88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff2e49fc4de google::LogMessage::Flush() at ??:0
@ 0x7ff2e49fda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff2e87607d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff2e874f73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff2fa42b1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff2fa42a783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff2fa4b62a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
I20260928 08:41:39.917896 17043 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7ff2fa4b60c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7ff2fa5005e9 std::function<>::operator()() at ??:0
@ 0x7ff2fa42ecdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff2fa42cc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff2fa4b4610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff2fa4b4439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff2e7648345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff2e764b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff2e764b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff2e7626eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff2ea9006db start_thread at ??:0
@ 0x7ff2e389661f clone at ??:0
W20260928 08:41:40.177249 17019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:41:40.177582 17019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:41:40.177763 17019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:41:40.185693 17019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:41:40.185842 17019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:41:40.186077 17019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:41:40.196811 17019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=33749
--tserver_master_addrs=127.0.97.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:41:40.198233 17019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:41:40.199853 17019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:41:40.216540 17067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:40.216490 17064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:40.216444 17065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:41:40.217470 17019 server_base.cc:1034] running on GCE node
I20260928 08:41:40.218223 17019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:41:40.219678 17019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:41:40.220992 17019 hybrid_clock.cc:648] HybridClock initialized: now 1790584900220932 us; error 85 us; skew 500 ppm
I20260928 08:41:40.221400 17019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:41:40 dist-test-slave-w6c0 krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:41:40.225049 17019 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:41:40.226804 17019 webserver.cc:460] Webserver started at http://127.0.97.1:33749/ using document root <none> and password file <none>
I20260928 08:41:40.227614 17019 fs_manager.cc:359] Metadata directory not provided
I20260928 08:41:40.227777 17019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:41:40.233857 17019 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 08:41:40.238819 17074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:40.241380 17019 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260928 08:41:40.241688 17019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "6a6185f2c3f747cd821eefd4e42d9880"
format_stamp: "Formatted at 2026-09-28 08:40:54 on dist-test-slave-w6c0"
server_key: "aa1d4bc936d8ffdef364e3eb5da6e187"
server_key_iv: "05b3147b670feb8bc4ab348d1cce1abf"
server_key_version: "encryptionkey@0"
I20260928 08:41:40.243060 17019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:41:40.264290 17019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:41:40.270494 17019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:41:40.270932 17019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:41:40.272158 17019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:41:40.274662 17081 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260928 08:41:40.281045 17019 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 08:41:40.281253 17019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260928 08:41:40.281422 17019 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 08:41:40.285576 17019 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 08:41:40.285779 17019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260928 08:41:40.286257 17081 tablet_bootstrap.cc:492] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880: Bootstrap starting.
I20260928 08:41:40.296819 17081 log.cc:824] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880: Log is configured to *not* fsync() on all Append() calls
I20260928 08:41:40.315750 17081 tablet_bootstrap.cc:492] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:41:40.317090 17081 tablet_bootstrap.cc:492] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880: Bootstrap complete.
I20260928 08:41:40.317986 17081 ts_tablet_manager.cc:1397] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880: Time spent bootstrapping tablet: real 0.032s user 0.029s sys 0.000s
I20260928 08:41:40.331017 17019 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45691
I20260928 08:41:40.331030 17188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45691 every 8 connection(s)
I20260928 08:41:40.333065 17019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:41:40.333485 17081 raft_consensus.cc:348] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } }
I20260928 08:41:40.335391 17081 raft_consensus.cc:729] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6185f2c3f747cd821eefd4e42d9880, State: Initialized, Role: FOLLOWER
I20260928 08:41:40.336381 17081 consensus_queue.cc:260] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } }
I20260928 08:41:40.338090 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 17019
I20260928 08:41:40.338617 17081 ts_tablet_manager.cc:1428] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880: Time spent starting tablet: real 0.020s user 0.017s sys 0.004s
I20260928 08:41:40.339140 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36155
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34279
--webserver_port=41923
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:41:40.340098 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:40.337188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:49097)
0928 08:41:40.338369 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:40.338713 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":493,"threads_started":1}
W20260928 08:41:40.340983 17189 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34279 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34279: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
I20260928 08:41:40.342274 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:40.341724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:48419)
0928 08:41:40.341968 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:40.342089 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 08:41:40.343400 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:40.342989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:41403)
0928 08:41:40.343173 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:40.343259 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 08:41:40.343874 17189 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:41:40.714033 17193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:41:40.714423 17193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:41:40.714618 17193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:41:40.722329 17193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:41:40.722555 17193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:41:40.722754 17193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:41:40.733178 17193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=41923
--tserver_master_addrs=127.0.97.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:41:40.734736 17193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:41:40.736161 17193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:41:40.751026 17198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:40.751363 17201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:40.751332 17199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:41:40.752095 17193 server_base.cc:1034] running on GCE node
I20260928 08:41:40.753624 17193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:41:40.755173 17193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:41:40.756489 17193 hybrid_clock.cc:648] HybridClock initialized: now 1790584900756406 us; error 85 us; skew 500 ppm
I20260928 08:41:40.756948 17193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:41:40 dist-test-slave-w6c0 krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:41:40.760619 17193 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:41:40.762601 17193 webserver.cc:460] Webserver started at http://127.0.97.2:41923/ using document root <none> and password file <none>
I20260928 08:41:40.763432 17193 fs_manager.cc:359] Metadata directory not provided
I20260928 08:41:40.763579 17193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:41:40.769518 17193 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 08:41:40.772590 17208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:40.774734 17193 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 08:41:40.775015 17193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "91a5b07dea91468881c22fdeb4281e05"
format_stamp: "Formatted at 2026-09-28 08:40:55 on dist-test-slave-w6c0"
server_key: "8bb41703a271c4f986871ddcfd31db31"
server_key_iv: "aec12794edd659d5a20f44817875be23"
server_key_version: "encryptionkey@0"
I20260928 08:41:40.776391 17193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:41:40.795292 17193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:41:40.801330 17193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:41:40.801793 17193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:41:40.803381 17193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:41:40.805410 17193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:41:40.805604 17193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:40.805774 17193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:41:40.805873 17193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:40.843304 17193 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36155
I20260928 08:41:40.843354 17320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36155 every 8 connection(s)
I20260928 08:41:40.844972 17193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:41:40.850327 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:40.848314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:36071)
0928 08:41:40.849225 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:40.849424 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":454,"threads_started":1}
I20260928 08:41:40.850864 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 17193
W20260928 08:41:40.851219 17321 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34279 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34279: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
I20260928 08:41:40.851716 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45131
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34279
--webserver_port=38389
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:41:40.852149 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:40.851800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:54421)
0928 08:41:40.851962 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:40.852042 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 08:41:40.853307 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:40.852965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:44045)
0928 08:41:40.853115 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:40.853200 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 08:41:40.853721 17321 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:41:41.236810 17324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:41:41.237226 17324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:41:41.237407 17324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:41:41.245807 17324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:41:41.246052 17324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:41:41.246260 17324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:41:41.257308 17324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38389
--tserver_master_addrs=127.0.97.62:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:41:41.258970 17324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:41:41.260453 17324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:41:41.275471 17332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:41.275436 17330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:41.275530 17329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:41:41.280431 17324 server_base.cc:1034] running on GCE node
I20260928 08:41:41.281185 17324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:41:41.282364 17324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:41:41.283572 17324 hybrid_clock.cc:648] HybridClock initialized: now 1790584901283511 us; error 58 us; skew 500 ppm
I20260928 08:41:41.283953 17324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:41:41 dist-test-slave-w6c0 krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:41:41.287565 17324 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:41:41.289273 17324 webserver.cc:460] Webserver started at http://127.0.97.3:38389/ using document root <none> and password file <none>
I20260928 08:41:41.290060 17324 fs_manager.cc:359] Metadata directory not provided
I20260928 08:41:41.290169 17324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:41:41.296144 17324 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 08:41:41.299222 17339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:41.301167 17324 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 08:41:41.301357 17324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "e491b4648bcd40bab94d37e206ebaa6a"
format_stamp: "Formatted at 2026-09-28 08:40:55 on dist-test-slave-w6c0"
server_key: "441fe1d7e9880e188ac4f5c169142826"
server_key_iv: "44d7cf3fd5639f9125a6e2f9bb40136a"
server_key_version: "encryptionkey@0"
I20260928 08:41:41.302436 17324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:41:41.339495 17324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:41:41.345119 17324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:41:41.345412 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:41.344539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:46323)
0928 08:41:41.345024 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:41.345116 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":251,"threads_started":1}
I20260928 08:41:41.345420 17324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:41:41.346599 17324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:41:41.348418 17324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:41:41.348582 17324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:41.348717 17324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:41:41.348821 17324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:41.390480 17324 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45131
I20260928 08:41:41.390671 17452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45131 every 8 connection(s)
I20260928 08:41:41.392632 17324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:41:41.398741 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:41.396520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:48499)
0928 08:41:41.397393 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:41.397619 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":359,"threads_started":1}
W20260928 08:41:41.399442 17453 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34279 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34279: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
I20260928 08:41:41.400403 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:41.400061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:45755)
0928 08:41:41.400225 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:41.400294 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 08:41:41.401229 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:41.400925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:60205)
0928 08:41:41.401046 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:41.401124 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 08:41:41.401558 17453 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 08:41:41.401880 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 17324
I20260928 08:41:41.708202 17457 raft_consensus.cc:482] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:41:41.708575 17457 raft_consensus.cc:504] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } }
I20260928 08:41:41.710487 17457 leader_election.cc:288] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91a5b07dea91468881c22fdeb4281e05 (127.0.97.2:36155), e491b4648bcd40bab94d37e206ebaa6a (127.0.97.3:45131)
Sep 28 08:41:41 dist-test-slave-w6c0 krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Sep 28 08:41:41 dist-test-slave-w6c0 krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:41:41.733135 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:41.711223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42117 (local address 127.0.97.2:36155)
0928 08:41:41.711902 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:41.711919 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:41.712923 (+ 1004us) server_negotiation.cc:407] Connection header received
0928 08:41:41.714701 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:41.714710 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:41.715035 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:41.715355 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:41:41.716852 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:41.718421 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:41.720384 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:41.721065 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:41.721283 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:41.725494 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:41:41.725504 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:41:41.725522 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:41:41.725564 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:41:41.728853 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:41.729958 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:41.729976 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:41.729982 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:41.730156 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:41.730676 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:41.730684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:41.730688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:41.731268 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:41:41.731478 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:41.732305 (+ 827us) server_negotiation.cc:299] Negotiation successful
0928 08:41:41.732722 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":349,"threads_started":1}
I20260928 08:41:41.733152 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:41.710892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36155 (local address 127.0.97.1:42117)
0928 08:41:41.712783 (+ 1891us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:41.712799 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:41:41.712964 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:41:41.715305 (+ 2341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:41:41.715312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:41:41.716012 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:41:41.716635 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:41.716656 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:41.718650 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:41.718656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:41.720227 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:41.720239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:41.721290 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:41.721297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:41.721665 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:41.722573 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:41:41.722592 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:41:41.725140 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:41:41.729040 (+ 3900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:41:41.729051 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:41:41.729068 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:41:41.729767 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:41:41.730299 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:41:41.730303 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:41:41.730305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:41:41.730549 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:41:41.731428 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:41:41.731437 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:41:41.731758 (+ 321us) client_negotiation.cc:769] Sending connection context
0928 08:41:41.732064 (+ 306us) client_negotiation.cc:240] Negotiation successful
0928 08:41:41.732638 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1755,"mutex_wait_us":1141,"spinlock_wait_cycles":16768,"thread_start_us":588,"threads_started":1}
I20260928 08:41:41.733211 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:41.710868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45131 (local address 127.0.97.1:40795)
0928 08:41:41.711059 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:41.711108 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:41:41.711691 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:41:41.715083 (+ 3392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:41:41.715106 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:41:41.715904 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:41:41.716634 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:41.716656 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:41.718738 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:41.718742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:41.720223 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:41.720237 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:41.721343 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:41.721351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:41.721635 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:41.722454 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:41:41.722482 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:41:41.725570 (+ 3088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:41:41.730204 (+ 4634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:41:41.730212 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:41:41.730218 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:41:41.730721 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:41:41.731173 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:41:41.731180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:41:41.731189 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:41:41.731310 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:41:41.732048 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:41:41.732052 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:41:41.732230 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 08:41:41.732389 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 08:41:41.732811 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":196}
I20260928 08:41:41.733536 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:41.711264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40795 (local address 127.0.97.3:45131)
0928 08:41:41.712407 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:41.712417 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:41.712456 (+ 39us) server_negotiation.cc:407] Connection header received
0928 08:41:41.714043 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:41.714053 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:41.714487 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:41.714943 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:41:41.716858 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:41.718493 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:41.720383 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:41.721067 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:41.721291 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:41.725875 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:41:41.725882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:41:41.725895 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:41:41.725944 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:41:41.729982 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:41.730860 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:41.730868 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:41.730877 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:41.731045 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:41.731423 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:41.731428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:41.731432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:41.731906 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:41:41.732088 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:41.732652 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 08:41:41.733077 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":993,"thread_start_us":839,"threads_started":1}
W20260928 08:41:41.736776 17078 leader_election.cc:341] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e491b4648bcd40bab94d37e206ebaa6a (127.0.97.3:45131): Not found: Tablet not found: 1a5341713efc434b86c821b6ca69ecca
W20260928 08:41:41.737027 17077 leader_election.cc:341] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91a5b07dea91468881c22fdeb4281e05 (127.0.97.2:36155): Not found: Tablet not found: 1a5341713efc434b86c821b6ca69ecca
I20260928 08:41:41.737241 17078 leader_election.cc:302] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a6185f2c3f747cd821eefd4e42d9880; no voters: 91a5b07dea91468881c22fdeb4281e05, e491b4648bcd40bab94d37e206ebaa6a
I20260928 08:41:41.737756 17457 raft_consensus.cc:2726] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:41:41.855469 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:41.854414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:47009)
0928 08:41:41.855029 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:41.855186 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":329,"threads_started":1}
I20260928 08:41:42.347311 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:42.346483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:51527)
0928 08:41:42.347044 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:42.347146 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":217,"threads_started":1}
I20260928 08:41:42.402962 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:42.402128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:45489)
0928 08:41:42.402638 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:42.402743 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":206,"threads_started":1}
I20260928 08:41:42.857122 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:42.856374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:35295)
0928 08:41:42.856870 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:42.856947 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":292,"threads_started":1}
I20260928 08:41:43.349025 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:43.348227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:33731)
0928 08:41:43.348746 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:43.348837 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":222,"threads_started":1}
I20260928 08:41:43.405094 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:43.404192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:42597)
0928 08:41:43.404794 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:43.404899 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":329,"threads_started":1}
I20260928 08:41:43.858762 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:43.858000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:50841)
0928 08:41:43.858436 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:43.858591 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":226,"threads_started":1}
I20260928 08:41:43.950174 17468 raft_consensus.cc:482] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:41:43.950388 17468 raft_consensus.cc:504] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } }
I20260928 08:41:43.951088 17468 leader_election.cc:288] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91a5b07dea91468881c22fdeb4281e05 (127.0.97.2:36155), e491b4648bcd40bab94d37e206ebaa6a (127.0.97.3:45131)
W20260928 08:41:43.952111 17078 leader_election.cc:341] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e491b4648bcd40bab94d37e206ebaa6a (127.0.97.3:45131): Not found: Tablet not found: 1a5341713efc434b86c821b6ca69ecca
W20260928 08:41:43.952283 17077 leader_election.cc:341] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91a5b07dea91468881c22fdeb4281e05 (127.0.97.2:36155): Not found: Tablet not found: 1a5341713efc434b86c821b6ca69ecca
I20260928 08:41:43.952463 17077 leader_election.cc:302] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a6185f2c3f747cd821eefd4e42d9880; no voters: 91a5b07dea91468881c22fdeb4281e05, e491b4648bcd40bab94d37e206ebaa6a
I20260928 08:41:43.952747 17468 raft_consensus.cc:2726] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:41:44.350893 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:44.349950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:52005)
0928 08:41:44.350574 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:44.350710 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":288,"threads_started":1}
I20260928 08:41:44.406996 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:44.406214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:54353)
0928 08:41:44.406703 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:44.406780 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":234,"threads_started":1}
I20260928 08:41:44.860348 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:44.859637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:58167)
0928 08:41:44.860092 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:44.860168 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":224,"threads_started":1}
I20260928 08:41:45.352735 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:45.351965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:48407)
0928 08:41:45.352450 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:45.352554 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":248,"threads_started":1}
I20260928 08:41:45.409075 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:45.408150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:39037)
0928 08:41:45.408734 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:45.408848 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":303,"threads_started":1}
I20260928 08:41:45.862052 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:45.861356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:42231)
0928 08:41:45.861802 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:45.861886 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":237,"threads_started":1}
I20260928 08:41:46.270172 17475 raft_consensus.cc:482] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:41:46.270368 17475 raft_consensus.cc:504] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } }
I20260928 08:41:46.271073 17475 leader_election.cc:288] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91a5b07dea91468881c22fdeb4281e05 (127.0.97.2:36155), e491b4648bcd40bab94d37e206ebaa6a (127.0.97.3:45131)
W20260928 08:41:46.272071 17077 leader_election.cc:341] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91a5b07dea91468881c22fdeb4281e05 (127.0.97.2:36155): Not found: Tablet not found: 1a5341713efc434b86c821b6ca69ecca
W20260928 08:41:46.272250 17078 leader_election.cc:341] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e491b4648bcd40bab94d37e206ebaa6a (127.0.97.3:45131): Not found: Tablet not found: 1a5341713efc434b86c821b6ca69ecca
I20260928 08:41:46.272403 17078 leader_election.cc:302] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a6185f2c3f747cd821eefd4e42d9880; no voters: 91a5b07dea91468881c22fdeb4281e05, e491b4648bcd40bab94d37e206ebaa6a
I20260928 08:41:46.272701 17475 raft_consensus.cc:2726] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:41:46.354713 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:46.353812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:57145)
0928 08:41:46.354315 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:46.354415 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":255,"threads_started":1}
I20260928 08:41:46.410945 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:46.410144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:43605)
0928 08:41:46.410686 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:46.410778 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":239,"threads_started":1}
I20260928 08:41:46.863911 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:46.863144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:57061)
0928 08:41:46.863644 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:46.863731 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":279,"threads_started":1}
I20260928 08:41:47.356403 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:47.355686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:36193)
0928 08:41:47.356110 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:47.356198 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":220,"threads_started":1}
I20260928 08:41:47.412485 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:47.411789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:46135)
0928 08:41:47.412235 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:47.412312 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":241,"threads_started":1}
I20260928 08:41:47.865729 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:47.864924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:58319)
0928 08:41:47.865435 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:47.865526 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":232,"threads_started":1}
I20260928 08:41:48.357986 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:48.357290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:33035)
0928 08:41:48.357731 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:48.357810 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":234,"threads_started":1}
I20260928 08:41:48.414146 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:48.413343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:42739)
0928 08:41:48.413860 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:48.413937 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":242,"threads_started":1}
I20260928 08:41:48.867617 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:48.866816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:45449)
0928 08:41:48.867335 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:48.867411 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":301,"threads_started":1}
I20260928 08:41:49.359901 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:49.359067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:44799)
0928 08:41:49.359590 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:49.359706 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":276,"threads_started":1}
I20260928 08:41:49.416090 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:49.415186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:35581)
0928 08:41:49.415745 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:49.415848 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":247,"threads_started":1}
I20260928 08:41:49.869206 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:49.868496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:57721)
0928 08:41:49.868966 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:49.869046 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":221,"threads_started":1}
I20260928 08:41:49.936131 17488 raft_consensus.cc:482] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:41:49.936386 17488 raft_consensus.cc:504] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } }
I20260928 08:41:49.937080 17488 leader_election.cc:288] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91a5b07dea91468881c22fdeb4281e05 (127.0.97.2:36155), e491b4648bcd40bab94d37e206ebaa6a (127.0.97.3:45131)
W20260928 08:41:49.938035 17078 leader_election.cc:341] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e491b4648bcd40bab94d37e206ebaa6a (127.0.97.3:45131): Not found: Tablet not found: 1a5341713efc434b86c821b6ca69ecca
W20260928 08:41:49.938247 17077 leader_election.cc:341] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91a5b07dea91468881c22fdeb4281e05 (127.0.97.2:36155): Not found: Tablet not found: 1a5341713efc434b86c821b6ca69ecca
I20260928 08:41:49.938390 17077 leader_election.cc:302] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a6185f2c3f747cd821eefd4e42d9880; no voters: 91a5b07dea91468881c22fdeb4281e05, e491b4648bcd40bab94d37e206ebaa6a
I20260928 08:41:49.938687 17488 raft_consensus.cc:2726] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:41:50.361717 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:50.360904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:46881)
0928 08:41:50.361429 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:50.361516 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":265,"threads_started":1}
I20260928 08:41:50.417805 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:50.416982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:45625)
0928 08:41:50.417556 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:50.417646 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":306,"threads_started":1}
I20260928 08:41:50.870947 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:50.870170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:35749)
0928 08:41:50.870654 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:50.870736 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":222,"threads_started":1}
I20260928 08:41:51.363512 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:51.362790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:44393)
0928 08:41:51.363240 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:51.363321 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":246,"threads_started":1}
I20260928 08:41:51.419512 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:51.418788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:54599)
0928 08:41:51.419245 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:51.419324 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":254,"threads_started":1}
I20260928 08:41:51.872749 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:51.871955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:49993)
0928 08:41:51.872492 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:51.872584 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":273,"threads_started":1}
I20260928 08:41:52.270088 17495 raft_consensus.cc:482] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:41:52.270278 17495 raft_consensus.cc:504] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a5b07dea91468881c22fdeb4281e05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36155 } } peers { permanent_uuid: "e491b4648bcd40bab94d37e206ebaa6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45131 } } peers { permanent_uuid: "6a6185f2c3f747cd821eefd4e42d9880" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45691 } }
I20260928 08:41:52.271013 17495 leader_election.cc:288] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91a5b07dea91468881c22fdeb4281e05 (127.0.97.2:36155), e491b4648bcd40bab94d37e206ebaa6a (127.0.97.3:45131)
W20260928 08:41:52.271978 17077 leader_election.cc:341] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91a5b07dea91468881c22fdeb4281e05 (127.0.97.2:36155): Not found: Tablet not found: 1a5341713efc434b86c821b6ca69ecca
W20260928 08:41:52.272166 17078 leader_election.cc:341] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e491b4648bcd40bab94d37e206ebaa6a (127.0.97.3:45131): Not found: Tablet not found: 1a5341713efc434b86c821b6ca69ecca
I20260928 08:41:52.272254 17078 leader_election.cc:302] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a6185f2c3f747cd821eefd4e42d9880; no voters: 91a5b07dea91468881c22fdeb4281e05, e491b4648bcd40bab94d37e206ebaa6a
I20260928 08:41:52.272475 17495 raft_consensus.cc:2726] T 1a5341713efc434b86c821b6ca69ecca P 6a6185f2c3f747cd821eefd4e42d9880 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:41:52.365520 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:52.364654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:40677)
0928 08:41:52.365164 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:52.365256 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":241,"threads_started":1}
I20260928 08:41:52.421571 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:52.420772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:33489)
0928 08:41:52.421293 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:52.421408 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":254,"threads_started":1}
I20260928 08:41:52.874372 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:52.873692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:58745)
0928 08:41:52.874117 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:52.874195 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":220,"threads_started":1}
I20260928 08:41:53.367477 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:53.366609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:33307)
0928 08:41:53.367156 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:53.367303 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":323,"threads_started":1}
I20260928 08:41:53.423744 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:53.422836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:49949)
0928 08:41:53.423451 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:53.423535 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":419,"threads_started":1}
I20260928 08:41:53.876032 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:53.875295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:57639)
0928 08:41:53.875742 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:53.875836 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":237,"threads_started":1}
I20260928 08:41:54.369223 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:54.368448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:44377)
0928 08:41:54.368946 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:54.369037 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":277,"threads_started":1}
I20260928 08:41:54.425496 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:54.424739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:34749)
0928 08:41:54.425242 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:54.425328 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":256,"threads_started":1}
I20260928 08:41:54.877516 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:54.876822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:52095)
0928 08:41:54.877253 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:54.877325 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":227,"threads_started":1}
I20260928 08:41:55.371038 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:55.370291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:48889)
0928 08:41:55.370747 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:55.370837 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":264,"threads_started":1}
I20260928 08:41:55.427136 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:55.426446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:38917)
0928 08:41:55.426880 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:55.426950 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":261,"threads_started":1}
I20260928 08:41:55.879380 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:55.878664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.2:46689)
0928 08:41:55.879124 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:55.879219 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":222,"threads_started":1}
I20260928 08:41:56.372668 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:56.371956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.1:57307)
0928 08:41:56.372413 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:56.372488 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":260,"threads_started":1}
I20260928 08:41:56.404403 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 17019
I20260928 08:41:56.420204 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:56.405396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37602 (local address 127.0.97.1:45691)
0928 08:41:56.406097 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:56.406105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:56.406130 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:41:56.406192 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:56.406198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:56.406459 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:56.406679 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:41:56.408451 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:56.409588 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:56.410896 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:56.411460 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:56.411611 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:56.414303 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:41:56.414313 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:41:56.414327 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:41:56.414357 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:41:56.417283 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:56.418032 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:56.418039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:56.418041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:56.418178 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:56.418622 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:56.418629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:56.418632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:56.418966 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:41:56.419295 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:56.419617 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 08:41:56.419873 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":373,"threads_started":1}
I20260928 08:41:56.421491 17184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:41:56.428831 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:56.428164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34279 (local address 127.0.97.3:54935)
0928 08:41:56.428586 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:56.428660 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":212,"threads_started":1}
I20260928 08:41:56.511976 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 17019
I20260928 08:41:56.531011 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 17193
I20260928 08:41:56.543342 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:56.531766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59020 (local address 127.0.97.2:36155)
0928 08:41:56.532198 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:56.532203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:56.532218 (+ 15us) server_negotiation.cc:407] Connection header received
0928 08:41:56.532273 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:56.532278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:56.532335 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:56.532423 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:41:56.533633 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:56.534389 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:56.535498 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:56.535880 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:56.535992 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:56.538387 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:41:56.538393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:41:56.538396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:41:56.538425 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:41:56.541038 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:56.541580 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:56.541584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:56.541587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:56.541699 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:56.542006 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:56.542009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:56.542011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:56.542250 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:41:56.542461 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:56.542757 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 08:41:56.542911 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":240,"threads_started":1}
I20260928 08:41:56.543994 17316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:41:56.627763 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 17193
I20260928 08:41:56.645560 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 17324
I20260928 08:41:56.658401 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:56.646423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41628 (local address 127.0.97.3:45131)
0928 08:41:56.647104 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:56.647109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:56.647125 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:41:56.647185 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:56.647189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:56.647252 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:56.647339 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:41:56.648661 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:56.649446 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:56.650684 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:56.651093 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:56.651188 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:56.653600 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:41:56.653605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:41:56.653608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:41:56.653639 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:41:56.656163 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:56.656748 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:56.656751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:56.656753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:56.656861 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:56.657175 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:56.657178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:56.657180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:56.657433 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:41:56.657627 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:56.657834 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 08:41:56.658001 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":383,"threads_started":1}
I20260928 08:41:56.659137 17448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:41:56.742769 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 17324
2026-09-28T08:41:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64354 ms)
[----------] 30 tests from SecurityITest (298151 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:41:56 dist-test-slave-w6c0 krb5kdc[17518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:41:56 dist-test-slave-w6c0 krb5kdc[17518](info): set up 2 sockets
Sep 28 08:41:56 dist-test-slave-w6c0 krb5kdc[17518](info): commencing operation
krb5kdc: starting...
W20260928 08:41:57.854578 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:41:58 dist-test-slave-w6c0 krb5kdc[17518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:41:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:41:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:41:58.102768 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42747
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:41:58.438683 17531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:41:58.438966 17531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:41:58.439055 17531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:41:58.446285 17531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:41:58.446388 17531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:41:58.446435 17531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:41:58.446492 17531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:41:58.446566 17531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:41:58.456425 17531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:41:58.457702 17531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:41:58.459018 17531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:41:58.470994 17536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:58.471319 17537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:58.470971 17539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:41:58.471572 17531 server_base.cc:1034] running on GCE node
I20260928 08:41:58.472388 17531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:41:58.473757 17531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:41:58.474983 17531 hybrid_clock.cc:648] HybridClock initialized: now 1790584918474945 us; error 52 us; skew 500 ppm
I20260928 08:41:58.475335 17531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:41:58 dist-test-slave-w6c0 krb5kdc[17518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:41:58.478340 17531 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:41:58.479789 17531 webserver.cc:460] Webserver started at http://127.0.97.62:36151/ using document root <none> and password file <none>
I20260928 08:41:58.480484 17531 fs_manager.cc:359] Metadata directory not provided
I20260928 08:41:58.480631 17531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:41:58.480957 17531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:41:58.484113 17531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "95db7110595e4a318dce45a9182cd8f4"
format_stamp: "Formatted at 2026-09-28 08:41:58 on dist-test-slave-w6c0"
server_key: "df4378b5a5b98ee8d6563ef4beaca97e"
server_key_iv: "db76393914b86b8f13cacbd550239f7c"
server_key_version: "encryptionkey@0"
I20260928 08:41:58.484890 17531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "95db7110595e4a318dce45a9182cd8f4"
format_stamp: "Formatted at 2026-09-28 08:41:58 on dist-test-slave-w6c0"
server_key: "df4378b5a5b98ee8d6563ef4beaca97e"
server_key_iv: "db76393914b86b8f13cacbd550239f7c"
server_key_version: "encryptionkey@0"
I20260928 08:41:58.490223 17531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:41:58.493726 17546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:58.495114 17531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:41:58.495309 17531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "95db7110595e4a318dce45a9182cd8f4"
format_stamp: "Formatted at 2026-09-28 08:41:58 on dist-test-slave-w6c0"
server_key: "df4378b5a5b98ee8d6563ef4beaca97e"
server_key_iv: "db76393914b86b8f13cacbd550239f7c"
server_key_version: "encryptionkey@0"
I20260928 08:41:58.495520 17531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:41:58.518654 17531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:41:58.522712 17531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:41:58.523012 17531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:41:58.541389 17531 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42747
I20260928 08:41:58.541430 17597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42747 every 8 connection(s)
I20260928 08:41:58.542794 17531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:41:58.546430 17598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:41:58.546576 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 17531
I20260928 08:41:58.546795 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:41:58.547453 388 external_mini_cluster.cc:1428] Setting key f569529f8f93a4c2fc7c14de94868354
I20260928 08:41:58.560750 17598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4: Bootstrap starting.
Sep 28 08:41:58 dist-test-slave-w6c0 krb5kdc[17518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:41:58.564018 17598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4: Neither blocks nor log segments found. Creating new log.
I20260928 08:41:58.565394 17598 log.cc:824] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4: Log is configured to *not* fsync() on all Append() calls
I20260928 08:41:58.568802 17598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4: No bootstrap required, opened a new log
I20260928 08:41:58.572948 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:58.549482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33384 (local address 127.0.97.62:42747)
0928 08:41:58.550290 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:58.550309 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:58.550363 (+ 54us) server_negotiation.cc:407] Connection header received
0928 08:41:58.553120 (+ 2757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:58.553150 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:58.553659 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:58.554101 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:41:58.555583 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:58.557593 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:58.559055 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:58.560066 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:58.560249 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:58.564432 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:41:58.564446 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:41:58.564471 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:41:58.564543 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:41:58.568460 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:58.569200 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:58.569211 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:58.569225 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:58.569449 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:58.569835 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:58.569840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:58.569842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:58.570529 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:41:58.570860 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:58.571168 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 08:41:58.571631 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":335,"threads_started":1}
I20260928 08:41:58.578194 17598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95db7110595e4a318dce45a9182cd8f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42747 } }
I20260928 08:41:58.578567 17598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:41:58.578651 17598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95db7110595e4a318dce45a9182cd8f4, State: Initialized, Role: FOLLOWER
I20260928 08:41:58.579183 17598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95db7110595e4a318dce45a9182cd8f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42747 } }
I20260928 08:41:58.579391 17598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:41:58.579486 17598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:41:58.579607 17598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:41:58.581717 17598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95db7110595e4a318dce45a9182cd8f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42747 } }
I20260928 08:41:58.582185 17598 leader_election.cc:302] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95db7110595e4a318dce45a9182cd8f4; no voters:
I20260928 08:41:58.582811 17598 leader_election.cc:288] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:41:58.582978 17603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:41:58.583698 17603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4 [term 1 LEADER]: Becoming Leader. State: Replica: 95db7110595e4a318dce45a9182cd8f4, State: Running, Role: LEADER
I20260928 08:41:58.584221 17603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95db7110595e4a318dce45a9182cd8f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42747 } }
I20260928 08:41:58.584913 17598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:41:58.587198 17604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95db7110595e4a318dce45a9182cd8f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95db7110595e4a318dce45a9182cd8f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42747 } } }
I20260928 08:41:58.587576 17604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4 [sys.catalog]: This master's current role is: LEADER
I20260928 08:41:58.587345 17605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95db7110595e4a318dce45a9182cd8f4. Latest consensus state: current_term: 1 leader_uuid: "95db7110595e4a318dce45a9182cd8f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95db7110595e4a318dce45a9182cd8f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42747 } } }
I20260928 08:41:58.587759 17605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4 [sys.catalog]: This master's current role is: LEADER
I20260928 08:41:58.589865 17609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:41:58.595010 17609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:41:58.603704 17609 catalog_manager.cc:1347] Generated new cluster ID: f6c54b9625f3466e95aba28c408ef1c8
I20260928 08:41:58.603821 17609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:41:58.617203 17609 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:41:58.619072 17609 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:41:58.631522 17609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4: Generated new TSK 0
I20260928 08:41:58.632381 17609 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:41:58.703644 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42747
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:41:59.052047 17624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:41:59.052313 17624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:41:59.052480 17624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:41:59.059818 17624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:41:59.059927 17624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:41:59.060141 17624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:41:59.069903 17624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:41:59.071252 17624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:41:59.072628 17624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:41:59.086444 17630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:59.086475 17629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:59.086542 17632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:41:59.087571 17624 server_base.cc:1034] running on GCE node
I20260928 08:41:59.088164 17624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:41:59.089094 17624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:41:59.090330 17624 hybrid_clock.cc:648] HybridClock initialized: now 1790584919090252 us; error 87 us; skew 500 ppm
I20260928 08:41:59.090746 17624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:41:59 dist-test-slave-w6c0 krb5kdc[17518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:41:59.094120 17624 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:41:59.095665 17624 webserver.cc:460] Webserver started at http://127.0.97.1:34329/ using document root <none> and password file <none>
I20260928 08:41:59.096365 17624 fs_manager.cc:359] Metadata directory not provided
I20260928 08:41:59.096513 17624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:41:59.096840 17624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:41:59.099879 17624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "1882f6717380482186a4099e722dcbf2"
format_stamp: "Formatted at 2026-09-28 08:41:59 on dist-test-slave-w6c0"
server_key: "629eb226732077f32e0c1cb3533a7bb9"
server_key_iv: "997a32b6734247a106b9d38e18dc2b03"
server_key_version: "encryptionkey@0"
I20260928 08:41:59.100708 17624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1882f6717380482186a4099e722dcbf2"
format_stamp: "Formatted at 2026-09-28 08:41:59 on dist-test-slave-w6c0"
server_key: "629eb226732077f32e0c1cb3533a7bb9"
server_key_iv: "997a32b6734247a106b9d38e18dc2b03"
server_key_version: "encryptionkey@0"
I20260928 08:41:59.105705 17624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260928 08:41:59.109073 17639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:59.110396 17624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 08:41:59.110591 17624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "1882f6717380482186a4099e722dcbf2"
format_stamp: "Formatted at 2026-09-28 08:41:59 on dist-test-slave-w6c0"
server_key: "629eb226732077f32e0c1cb3533a7bb9"
server_key_iv: "997a32b6734247a106b9d38e18dc2b03"
server_key_version: "encryptionkey@0"
I20260928 08:41:59.110764 17624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:41:59.132552 17624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:41:59.136597 17624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:41:59.136831 17624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:41:59.137740 17624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:41:59.139019 17624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:41:59.139102 17624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:59.139195 17624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:41:59.139248 17624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:59.173275 17624 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44271
I20260928 08:41:59.173322 17751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44271 every 8 connection(s)
I20260928 08:41:59.174619 17624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:41:59.182690 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 17624
I20260928 08:41:59.182916 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:41:59.183665 388 external_mini_cluster.cc:1428] Setting key 48b4980c590a5dd90426369979105193
Sep 28 08:41:59 dist-test-slave-w6c0 krb5kdc[17518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:41:59.197481 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:59.177238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42745 (local address 127.0.97.62:42747)
0928 08:41:59.177715 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:59.177722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:59.178606 (+ 884us) server_negotiation.cc:407] Connection header received
0928 08:41:59.180384 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:59.180389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:59.180466 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:59.180565 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:41:59.182000 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:59.183249 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:59.185063 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:59.185582 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:59.185768 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:59.189217 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:41:59.189226 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:41:59.189234 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:41:59.189278 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:41:59.193056 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:59.194046 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:59.194055 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:59.194061 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:59.194231 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:59.194807 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:59.194812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:59.194816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:59.195295 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:41:59.195453 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:59.196641 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0928 08:41:59.196981 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":222,"threads_started":1}
I20260928 08:41:59.198405 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:59.177695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42747 (local address 127.0.97.1:42745)
0928 08:41:59.178401 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:59.178438 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:41:59.180143 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:41:59.180758 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:41:59.180766 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:41:59.181266 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:41:59.181829 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:59.181850 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:59.183415 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:59.183419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:59.184932 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:59.184946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:59.185880 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:59.185891 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:59.186106 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:59.186851 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:41:59.186873 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:41:59.188996 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:41:59.193185 (+ 4189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:41:59.193192 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:41:59.193204 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:41:59.193907 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:41:59.194365 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:41:59.194371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:41:59.194374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:41:59.194675 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:41:59.195455 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:41:59.195493 (+ 38us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:41:59.196035 (+ 542us) client_negotiation.cc:769] Sending connection context
0928 08:41:59.196446 (+ 411us) client_negotiation.cc:240] Negotiation successful
0928 08:41:59.196987 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":268,"threads_started":1}
I20260928 08:41:59.200059 17752 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42747
I20260928 08:41:59.200474 17752 heartbeater.cc:464] Registering TS with master...
I20260928 08:41:59.201362 17752 heartbeater.cc:511] Master 127.0.97.62:42747 requested a full tablet report, sending...
I20260928 08:41:59.203985 17563 ts_manager.cc:194] Registered new tserver with Master: 1882f6717380482186a4099e722dcbf2 (127.0.97.1:44271)
I20260928 08:41:59.206094 17563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42745
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:41:59.242156 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42747
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:41:59.568663 17758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:41:59.568954 17758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:41:59.569052 17758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:41:59.576292 17758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:41:59.576398 17758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:41:59.576512 17758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:41:59.586292 17758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:41:59.587535 17758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:41:59.588783 17758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:41:59.601984 17763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:59.602037 17764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:41:59.602061 17766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:41:59.603127 17758 server_base.cc:1034] running on GCE node
I20260928 08:41:59.603781 17758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:41:59.604619 17758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:41:59.605885 17758 hybrid_clock.cc:648] HybridClock initialized: now 1790584919605835 us; error 60 us; skew 500 ppm
I20260928 08:41:59.606210 17758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:41:59 dist-test-slave-w6c0 krb5kdc[17518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:41:59.615746 17758 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:41:59.617257 17758 webserver.cc:460] Webserver started at http://127.0.97.2:34457/ using document root <none> and password file <none>
I20260928 08:41:59.617945 17758 fs_manager.cc:359] Metadata directory not provided
I20260928 08:41:59.618093 17758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:41:59.618400 17758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:41:59.621387 17758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "8e0ebd64af7440c5bd40b1b10a73cd92"
format_stamp: "Formatted at 2026-09-28 08:41:59 on dist-test-slave-w6c0"
server_key: "57f3e940769a4ba943f6c032b120434d"
server_key_iv: "cd55bfff37ff98271a6d4ec4bbae257f"
server_key_version: "encryptionkey@0"
I20260928 08:41:59.622170 17758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8e0ebd64af7440c5bd40b1b10a73cd92"
format_stamp: "Formatted at 2026-09-28 08:41:59 on dist-test-slave-w6c0"
server_key: "57f3e940769a4ba943f6c032b120434d"
server_key_iv: "cd55bfff37ff98271a6d4ec4bbae257f"
server_key_version: "encryptionkey@0"
I20260928 08:41:59.627123 17758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 08:41:59.630425 17773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:59.631742 17758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 08:41:59.631924 17758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "8e0ebd64af7440c5bd40b1b10a73cd92"
format_stamp: "Formatted at 2026-09-28 08:41:59 on dist-test-slave-w6c0"
server_key: "57f3e940769a4ba943f6c032b120434d"
server_key_iv: "cd55bfff37ff98271a6d4ec4bbae257f"
server_key_version: "encryptionkey@0"
I20260928 08:41:59.632117 17758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:41:59.653260 17758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:41:59.657305 17758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:41:59.657598 17758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:41:59.658628 17758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:41:59.659963 17758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:41:59.660070 17758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:59.660215 17758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:41:59.660288 17758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:41:59.693590 17758 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43731
I20260928 08:41:59.693630 17885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43731 every 8 connection(s)
I20260928 08:41:59.694938 17758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:41:59.695946 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 17758
I20260928 08:41:59.696135 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:41:59.696650 388 external_mini_cluster.cc:1428] Setting key 7dd9c36a5cb0618369dcea189b0a6967
Sep 28 08:41:59 dist-test-slave-w6c0 krb5kdc[17518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:41:59.718684 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:59.697437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48159 (local address 127.0.97.62:42747)
0928 08:41:59.697633 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 08:41:59.697638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:41:59.698996 (+ 1358us) server_negotiation.cc:407] Connection header received
0928 08:41:59.701888 (+ 2892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:41:59.701892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:41:59.701958 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:41:59.702040 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:41:59.704109 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:59.704822 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:59.707324 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:41:59.707735 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:41:59.707831 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:59.711675 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:41:59.711684 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:41:59.711692 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:41:59.711738 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:41:59.715351 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:59.716085 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:59.716093 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:59.716099 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:59.716259 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:41:59.716691 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:41:59.716698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:41:59.716704 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:41:59.717103 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:41:59.717255 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 08:41:59.717934 (+ 679us) server_negotiation.cc:299] Negotiation successful
0928 08:41:59.718193 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:41:59.719370 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:41:59.698027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42747 (local address 127.0.97.2:48159)
0928 08:41:59.698808 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0928 08:41:59.698851 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:41:59.701597 (+ 2746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:41:59.702325 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:41:59.702339 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:41:59.703118 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:41:59.703926 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:59.703953 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:59.704984 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:59.704993 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:59.707168 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:41:59.707186 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:41:59.707969 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:41:59.707978 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:41:59.708316 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:41:59.709101 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:41:59.709128 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:41:59.711414 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:41:59.715482 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:41:59.715489 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:41:59.715501 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:41:59.715922 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:41:59.716378 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:41:59.716381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:41:59.716383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:41:59.716559 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:41:59.717222 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:41:59.717234 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:41:59.717546 (+ 312us) client_negotiation.cc:769] Sending connection context
0928 08:41:59.717773 (+ 227us) client_negotiation.cc:240] Negotiation successful
0928 08:41:59.718165 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":294,"threads_started":1}
I20260928 08:41:59.720932 17886 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42747
I20260928 08:41:59.721307 17886 heartbeater.cc:464] Registering TS with master...
I20260928 08:41:59.722213 17886 heartbeater.cc:511] Master 127.0.97.62:42747 requested a full tablet report, sending...
I20260928 08:41:59.723987 17563 ts_manager.cc:194] Registered new tserver with Master: 8e0ebd64af7440c5bd40b1b10a73cd92 (127.0.97.2:43731)
I20260928 08:41:59.724854 17563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48159
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:41:59.755771 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42747
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:00.079614 17891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:00.079890 17891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:00.080045 17891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:00.087735 17891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:00.087834 17891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:00.088001 17891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:42:00.098039 17891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:00.099418 17891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:00.100751 17891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:00.113787 17896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:00.113812 17897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:00.114020 17899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:00.114717 17891 server_base.cc:1034] running on GCE node
I20260928 08:42:00.115368 17891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:00.116250 17891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:00.117522 17891 hybrid_clock.cc:648] HybridClock initialized: now 1790584920117458 us; error 37 us; skew 500 ppm
I20260928 08:42:00.117849 17891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:00 dist-test-slave-w6c0 krb5kdc[17518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:00.121191 17891 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:42:00.122776 17891 webserver.cc:460] Webserver started at http://127.0.97.3:38547/ using document root <none> and password file <none>
I20260928 08:42:00.123479 17891 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:00.123615 17891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:00.123932 17891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:00.126884 17891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "d48b7b5ba59b42f7b2b41b8e6e1e5cb5"
format_stamp: "Formatted at 2026-09-28 08:42:00 on dist-test-slave-w6c0"
server_key: "f68d2cdd11c72d051e36383862af7e1e"
server_key_iv: "1710c1dfd04a5055db8723957be78552"
server_key_version: "encryptionkey@0"
I20260928 08:42:00.127702 17891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d48b7b5ba59b42f7b2b41b8e6e1e5cb5"
format_stamp: "Formatted at 2026-09-28 08:42:00 on dist-test-slave-w6c0"
server_key: "f68d2cdd11c72d051e36383862af7e1e"
server_key_iv: "1710c1dfd04a5055db8723957be78552"
server_key_version: "encryptionkey@0"
I20260928 08:42:00.132885 17891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:42:00.136698 17906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:00.138069 17891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:42:00.138257 17891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "d48b7b5ba59b42f7b2b41b8e6e1e5cb5"
format_stamp: "Formatted at 2026-09-28 08:42:00 on dist-test-slave-w6c0"
server_key: "f68d2cdd11c72d051e36383862af7e1e"
server_key_iv: "1710c1dfd04a5055db8723957be78552"
server_key_version: "encryptionkey@0"
I20260928 08:42:00.138459 17891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:00.158133 17891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:00.162253 17891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:00.162596 17891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:00.163551 17891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:42:00.164873 17891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:42:00.164980 17891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:00.165143 17891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:42:00.165215 17891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:00.199800 17891 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43995
I20260928 08:42:00.199903 18018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43995 every 8 connection(s)
I20260928 08:42:00.201076 17891 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:42:00.208599 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 17891
I20260928 08:42:00.208813 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:42:00.209245 17752 heartbeater.cc:503] Master 127.0.97.62:42747 was elected leader, sending a full tablet report...
I20260928 08:42:00.209421 388 external_mini_cluster.cc:1428] Setting key dca706f73bed072f341c121248855434
Sep 28 08:42:00 dist-test-slave-w6c0 krb5kdc[17518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:42:00.221169 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.203662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60953 (local address 127.0.97.62:42747)
0928 08:42:00.203923 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:00.203929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:00.204967 (+ 1038us) server_negotiation.cc:407] Connection header received
0928 08:42:00.206806 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:00.206811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:00.206884 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:00.206980 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:00.208542 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.209524 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.211771 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.212355 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.212450 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:00.215833 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:00.215838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:00.215842 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:00.215881 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:00.218275 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:00.219079 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:00.219082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:00.219085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:00.219216 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:00.219635 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:00.219638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:00.219640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:00.219954 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:00.220119 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:00.220748 (+ 629us) server_negotiation.cc:299] Negotiation successful
0928 08:42:00.220924 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260928 08:42:00.222155 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.204113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42747 (local address 127.0.97.3:60953)
0928 08:42:00.204802 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:00.204844 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:00.206459 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:00.207190 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:00.207198 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:00.207719 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:00.208355 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:00.208382 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.209743 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.209750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:00.211641 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:00.211653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.212580 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.212586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:00.212789 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:00.213404 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:42:00.213447 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:00.215650 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:00.218438 (+ 2788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:00.218444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:00.218457 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:00.218963 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:00.219340 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:00.219344 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:00.219346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:00.219530 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:00.220079 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:00.220084 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:00.220400 (+ 316us) client_negotiation.cc:769] Sending connection context
0928 08:42:00.220645 (+ 245us) client_negotiation.cc:240] Negotiation successful
0928 08:42:00.221049 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":272,"threads_started":1}
I20260928 08:42:00.223779 18019 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42747
I20260928 08:42:00.224140 18019 heartbeater.cc:464] Registering TS with master...
I20260928 08:42:00.224902 18019 heartbeater.cc:511] Master 127.0.97.62:42747 requested a full tablet report, sending...
I20260928 08:42:00.226414 17563 ts_manager.cc:194] Registered new tserver with Master: d48b7b5ba59b42f7b2b41b8e6e1e5cb5 (127.0.97.3:43995)
I20260928 08:42:00.227276 17563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60953
I20260928 08:42:00.237103 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:42:00.254832 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.241660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.0.97.62:42747)
0928 08:42:00.241918 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:00.241925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:00.242268 (+ 343us) server_negotiation.cc:407] Connection header received
0928 08:42:00.242482 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:00.242487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:00.242635 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:00.242745 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:00.244333 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.245086 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.246430 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.246939 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.247086 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:00.249483 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:00.249489 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:00.249496 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:00.249535 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:00.252105 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:00.252758 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:00.252765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:00.252769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:00.252896 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:00.253283 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:00.253291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:00.253294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:00.253636 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:00.253896 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:00.254247 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 08:42:00.254479 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 08:42:00.258634 17563 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:42:00.265256 17563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:42:00.306703 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.277385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43995 (local address 127.0.0.1:47926)
0928 08:42:00.298028 (+ 20643us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:00.298053 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:00.298200 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:00.299103 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:00.299114 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:00.299141 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:00.299543 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:00.299552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.301188 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.301194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:00.303731 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:00.303744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.305559 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.305567 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:00.305761 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:00.305818 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 08:42:00.305956 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 08:42:00.306147 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20500,"mutex_wait_us":238,"thread_start_us":20363,"threads_started":1}
I20260928 08:42:00.307219 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.276734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.0.1:39960)
0928 08:42:00.289895 (+ 13161us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:00.289939 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:00.290175 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:00.298264 (+ 8089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:00.298271 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:00.298304 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:00.298860 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:00.298874 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.301255 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.301260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:00.304507 (+ 3247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:00.304521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.306274 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.306280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:00.306476 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:00.306562 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 08:42:00.306659 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 08:42:00.306799 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12875,"mutex_wait_us":183,"thread_start_us":12701,"threads_started":1}
I20260928 08:42:00.307775 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.277044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39960 (local address 127.0.97.1:44271)
0928 08:42:00.297504 (+ 20460us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:00.297512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:00.297544 (+ 32us) server_negotiation.cc:407] Connection header received
0928 08:42:00.297681 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:00.297693 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:00.297996 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:00.299079 (+ 1083us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:00.299328 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.301087 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.304693 (+ 3606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.306062 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.306214 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:00.306442 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:00.306754 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 08:42:00.307279 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20290,"thread_start_us":20145,"threads_started":1}
I20260928 08:42:00.307871 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.277373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.0.97.2:43731)
0928 08:42:00.297671 (+ 20298us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:00.297684 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:00.297718 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:42:00.297826 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:00.297833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:00.298077 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:00.298282 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:00.299445 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.301098 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.304670 (+ 3572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.306059 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.306190 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:00.306375 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:00.307514 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0928 08:42:00.307642 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20072,"thread_start_us":20030,"threads_started":1}
I20260928 08:42:00.308276 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.276920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43731 (local address 127.0.0.1:39798)
0928 08:42:00.296688 (+ 19768us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:00.296717 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:00.296886 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:00.298589 (+ 1703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:00.298593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:00.298657 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:00.299233 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:00.299251 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.301266 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.301271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:00.304512 (+ 3241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:00.304521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.306711 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.306717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:00.306869 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:00.306918 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 08:42:00.306992 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 08:42:00.307121 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19559,"mutex_wait_us":297,"thread_start_us":19364,"threads_started":1}
I20260928 08:42:00.308549 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.278065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.0.97.3:43995)
0928 08:42:00.297117 (+ 19052us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:00.297131 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:00.298337 (+ 1206us) server_negotiation.cc:407] Connection header received
0928 08:42:00.298442 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:00.298451 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:00.298705 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:00.298889 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:00.299781 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.300980 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.303908 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.305035 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.305165 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:00.305397 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:00.308055 (+ 2658us) server_negotiation.cc:299] Negotiation successful
0928 08:42:00.308226 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18844,"thread_start_us":18663,"threads_started":1}
I20260928 08:42:00.311787 17954 tablet_service.cc:1467] Processing CreateTablet for tablet 170995d6a7ed4553b4bbdf89c784956c (DEFAULT_TABLE table=test-table [id=e1720f15b55843c6b36e6ed6acfc00cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:42:00.311887 17687 tablet_service.cc:1467] Processing CreateTablet for tablet 170995d6a7ed4553b4bbdf89c784956c (DEFAULT_TABLE table=test-table [id=e1720f15b55843c6b36e6ed6acfc00cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:42:00.311887 17821 tablet_service.cc:1467] Processing CreateTablet for tablet 170995d6a7ed4553b4bbdf89c784956c (DEFAULT_TABLE table=test-table [id=e1720f15b55843c6b36e6ed6acfc00cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:42:00.313162 17954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170995d6a7ed4553b4bbdf89c784956c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:00.313218 17821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170995d6a7ed4553b4bbdf89c784956c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:00.313217 17687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170995d6a7ed4553b4bbdf89c784956c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:00.323876 18037 tablet_bootstrap.cc:492] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5: Bootstrap starting.
I20260928 08:42:00.327273 18037 tablet_bootstrap.cc:654] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:00.328691 18037 log.cc:824] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:00.331084 18039 tablet_bootstrap.cc:492] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92: Bootstrap starting.
I20260928 08:42:00.331661 18037 tablet_bootstrap.cc:492] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5: No bootstrap required, opened a new log
I20260928 08:42:00.331915 18037 ts_tablet_manager.cc:1397] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 08:42:00.334198 18039 tablet_bootstrap.cc:654] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:00.334640 18038 tablet_bootstrap.cc:492] T 170995d6a7ed4553b4bbdf89c784956c P 1882f6717380482186a4099e722dcbf2: Bootstrap starting.
I20260928 08:42:00.335459 18039 log.cc:824] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:00.338168 18038 tablet_bootstrap.cc:654] T 170995d6a7ed4553b4bbdf89c784956c P 1882f6717380482186a4099e722dcbf2: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:00.338666 18039 tablet_bootstrap.cc:492] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92: No bootstrap required, opened a new log
I20260928 08:42:00.338932 18039 ts_tablet_manager.cc:1397] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260928 08:42:00.339567 18038 log.cc:824] T 170995d6a7ed4553b4bbdf89c784956c P 1882f6717380482186a4099e722dcbf2: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:00.341512 18037 raft_consensus.cc:348] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f6717380482186a4099e722dcbf2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } } peers { permanent_uuid: "8e0ebd64af7440c5bd40b1b10a73cd92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43731 } } peers { permanent_uuid: "d48b7b5ba59b42f7b2b41b8e6e1e5cb5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43995 } }
I20260928 08:42:00.341902 18037 raft_consensus.cc:374] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:00.342000 18037 raft_consensus.cc:729] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d48b7b5ba59b42f7b2b41b8e6e1e5cb5, State: Initialized, Role: FOLLOWER
I20260928 08:42:00.342686 18037 consensus_queue.cc:260] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f6717380482186a4099e722dcbf2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } } peers { permanent_uuid: "8e0ebd64af7440c5bd40b1b10a73cd92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43731 } } peers { permanent_uuid: "d48b7b5ba59b42f7b2b41b8e6e1e5cb5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43995 } }
I20260928 08:42:00.344129 18037 ts_tablet_manager.cc:1428] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260928 08:42:00.344597 18019 heartbeater.cc:503] Master 127.0.97.62:42747 was elected leader, sending a full tablet report...
I20260928 08:42:00.345137 18038 tablet_bootstrap.cc:492] T 170995d6a7ed4553b4bbdf89c784956c P 1882f6717380482186a4099e722dcbf2: No bootstrap required, opened a new log
I20260928 08:42:00.345396 18038 ts_tablet_manager.cc:1397] T 170995d6a7ed4553b4bbdf89c784956c P 1882f6717380482186a4099e722dcbf2: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260928 08:42:00.349998 18039 raft_consensus.cc:348] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f6717380482186a4099e722dcbf2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } } peers { permanent_uuid: "8e0ebd64af7440c5bd40b1b10a73cd92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43731 } } peers { permanent_uuid: "d48b7b5ba59b42f7b2b41b8e6e1e5cb5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43995 } }
I20260928 08:42:00.350445 18039 raft_consensus.cc:374] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:00.350590 18039 raft_consensus.cc:729] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0ebd64af7440c5bd40b1b10a73cd92, State: Initialized, Role: FOLLOWER
I20260928 08:42:00.350865 18043 raft_consensus.cc:482] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:42:00.351120 18043 raft_consensus.cc:504] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f6717380482186a4099e722dcbf2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } } peers { permanent_uuid: "8e0ebd64af7440c5bd40b1b10a73cd92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43731 } } peers { permanent_uuid: "d48b7b5ba59b42f7b2b41b8e6e1e5cb5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43995 } }
I20260928 08:42:00.351315 18039 consensus_queue.cc:260] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f6717380482186a4099e722dcbf2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } } peers { permanent_uuid: "8e0ebd64af7440c5bd40b1b10a73cd92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43731 } } peers { permanent_uuid: "d48b7b5ba59b42f7b2b41b8e6e1e5cb5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43995 } }
I20260928 08:42:00.352440 17886 heartbeater.cc:503] Master 127.0.97.62:42747 was elected leader, sending a full tablet report...
I20260928 08:42:00.353116 18039 ts_tablet_manager.cc:1428] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260928 08:42:00.356243 18038 raft_consensus.cc:348] T 170995d6a7ed4553b4bbdf89c784956c P 1882f6717380482186a4099e722dcbf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f6717380482186a4099e722dcbf2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } } peers { permanent_uuid: "8e0ebd64af7440c5bd40b1b10a73cd92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43731 } } peers { permanent_uuid: "d48b7b5ba59b42f7b2b41b8e6e1e5cb5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43995 } }
I20260928 08:42:00.356698 18038 raft_consensus.cc:374] T 170995d6a7ed4553b4bbdf89c784956c P 1882f6717380482186a4099e722dcbf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:00.356801 18038 raft_consensus.cc:729] T 170995d6a7ed4553b4bbdf89c784956c P 1882f6717380482186a4099e722dcbf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1882f6717380482186a4099e722dcbf2, State: Initialized, Role: FOLLOWER
I20260928 08:42:00.357533 18038 consensus_queue.cc:260] T 170995d6a7ed4553b4bbdf89c784956c P 1882f6717380482186a4099e722dcbf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f6717380482186a4099e722dcbf2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } } peers { permanent_uuid: "8e0ebd64af7440c5bd40b1b10a73cd92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43731 } } peers { permanent_uuid: "d48b7b5ba59b42f7b2b41b8e6e1e5cb5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43995 } }
I20260928 08:42:00.360347 18038 ts_tablet_manager.cc:1428] T 170995d6a7ed4553b4bbdf89c784956c P 1882f6717380482186a4099e722dcbf2: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260928 08:42:00.360916 18043 leader_election.cc:288] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1882f6717380482186a4099e722dcbf2 (127.0.97.1:44271), 8e0ebd64af7440c5bd40b1b10a73cd92 (127.0.97.2:43731)
I20260928 08:42:00.362130 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.353175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.97.3:44575)
0928 08:42:00.353549 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:00.353572 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:00.353701 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:00.354312 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:00.354316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:00.354342 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:00.354582 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:00.354594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.356484 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.356492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:00.359245 (+ 2753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:00.359266 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.361325 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.361332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:00.361549 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:00.361574 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 08:42:00.361690 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 08:42:00.361830 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
I20260928 08:42:00.363065 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.353652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44575 (local address 127.0.97.1:44271)
0928 08:42:00.353832 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:00.353838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:00.353856 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:42:00.353920 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:00.353927 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:00.354012 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:00.354121 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:00.354762 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.356283 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.359445 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.360913 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.361089 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:00.361231 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:00.362767 (+ 1536us) server_negotiation.cc:299] Negotiation successful
0928 08:42:00.362902 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 08:42:00.363883 17707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170995d6a7ed4553b4bbdf89c784956c" candidate_uuid: "d48b7b5ba59b42f7b2b41b8e6e1e5cb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1882f6717380482186a4099e722dcbf2" is_pre_election: true
I20260928 08:42:00.364336 17707 raft_consensus.cc:2454] T 170995d6a7ed4553b4bbdf89c784956c P 1882f6717380482186a4099e722dcbf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d48b7b5ba59b42f7b2b41b8e6e1e5cb5 in term 0.
I20260928 08:42:00.364955 17910 leader_election.cc:302] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1882f6717380482186a4099e722dcbf2, d48b7b5ba59b42f7b2b41b8e6e1e5cb5; no voters:
I20260928 08:42:00.365300 18043 raft_consensus.cc:2781] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:42:00.365468 18043 raft_consensus.cc:482] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:42:00.365586 18043 raft_consensus.cc:3037] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:00.366995 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.354478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40157 (local address 127.0.97.2:43731)
0928 08:42:00.354691 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:00.354699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:00.361759 (+ 7060us) server_negotiation.cc:407] Connection header received
0928 08:42:00.361947 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:00.361952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:00.362029 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:00.362128 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:00.362732 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.363522 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.365191 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.366148 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.366247 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:00.366309 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:00.366741 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 08:42:00.366830 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 08:42:00.367208 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.354150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43731 (local address 127.0.97.3:40157)
0928 08:42:00.361670 (+ 7520us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:00.361691 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:00.361814 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:00.362144 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:00.362149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:00.362172 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:00.362578 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:00.362591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.363635 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.363638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:00.365010 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:00.365022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.366358 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.366364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:00.366579 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:00.366620 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 08:42:00.366720 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 08:42:00.366843 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7188,"thread_start_us":7019,"threads_started":1}
I20260928 08:42:00.368068 17841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170995d6a7ed4553b4bbdf89c784956c" candidate_uuid: "d48b7b5ba59b42f7b2b41b8e6e1e5cb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0ebd64af7440c5bd40b1b10a73cd92" is_pre_election: true
I20260928 08:42:00.368498 17841 raft_consensus.cc:2454] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d48b7b5ba59b42f7b2b41b8e6e1e5cb5 in term 0.
I20260928 08:42:00.368523 18043 raft_consensus.cc:504] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f6717380482186a4099e722dcbf2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } } peers { permanent_uuid: "8e0ebd64af7440c5bd40b1b10a73cd92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43731 } } peers { permanent_uuid: "d48b7b5ba59b42f7b2b41b8e6e1e5cb5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43995 } }
I20260928 08:42:00.369204 18043 leader_election.cc:288] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [CANDIDATE]: Term 1 election: Requested vote from peers 1882f6717380482186a4099e722dcbf2 (127.0.97.1:44271), 8e0ebd64af7440c5bd40b1b10a73cd92 (127.0.97.2:43731)
I20260928 08:42:00.369644 17707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170995d6a7ed4553b4bbdf89c784956c" candidate_uuid: "d48b7b5ba59b42f7b2b41b8e6e1e5cb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1882f6717380482186a4099e722dcbf2"
I20260928 08:42:00.369655 17841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170995d6a7ed4553b4bbdf89c784956c" candidate_uuid: "d48b7b5ba59b42f7b2b41b8e6e1e5cb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0ebd64af7440c5bd40b1b10a73cd92"
I20260928 08:42:00.369838 17707 raft_consensus.cc:3037] T 170995d6a7ed4553b4bbdf89c784956c P 1882f6717380482186a4099e722dcbf2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:00.369846 17841 raft_consensus.cc:3037] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:00.372027 17841 raft_consensus.cc:2454] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d48b7b5ba59b42f7b2b41b8e6e1e5cb5 in term 1.
I20260928 08:42:00.372032 17707 raft_consensus.cc:2454] T 170995d6a7ed4553b4bbdf89c784956c P 1882f6717380482186a4099e722dcbf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d48b7b5ba59b42f7b2b41b8e6e1e5cb5 in term 1.
I20260928 08:42:00.372411 17910 leader_election.cc:302] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1882f6717380482186a4099e722dcbf2, d48b7b5ba59b42f7b2b41b8e6e1e5cb5; no voters:
I20260928 08:42:00.372794 18043 raft_consensus.cc:2781] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:00.373270 18043 raft_consensus.cc:686] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [term 1 LEADER]: Becoming Leader. State: Replica: d48b7b5ba59b42f7b2b41b8e6e1e5cb5, State: Running, Role: LEADER
I20260928 08:42:00.373797 18043 consensus_queue.cc:237] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f6717380482186a4099e722dcbf2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } } peers { permanent_uuid: "8e0ebd64af7440c5bd40b1b10a73cd92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43731 } } peers { permanent_uuid: "d48b7b5ba59b42f7b2b41b8e6e1e5cb5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43995 } }
I20260928 08:42:00.378139 17563 catalog_manager.cc:5495] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d48b7b5ba59b42f7b2b41b8e6e1e5cb5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d48b7b5ba59b42f7b2b41b8e6e1e5cb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f6717380482186a4099e722dcbf2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e0ebd64af7440c5bd40b1b10a73cd92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d48b7b5ba59b42f7b2b41b8e6e1e5cb5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43995 } health_report { overall_health: HEALTHY } } }
I20260928 08:42:00.400765 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.395243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.0.97.3:43995)
0928 08:42:00.395452 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:00.395458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:00.395478 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:42:00.395682 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:00.395686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:00.395762 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:00.395913 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:42:00.396293 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.397003 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.398462 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.398973 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.399120 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:00.399631 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:42:00.400183 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:42:00.400346 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:00.400418 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 08:42:00.400497 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:42:00.408532 17707 raft_consensus.cc:1261] T 170995d6a7ed4553b4bbdf89c784956c P 1882f6717380482186a4099e722dcbf2 [term 1 FOLLOWER]: Refusing update from remote peer d48b7b5ba59b42f7b2b41b8e6e1e5cb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:42:00.408535 17841 raft_consensus.cc:1261] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92 [term 1 FOLLOWER]: Refusing update from remote peer d48b7b5ba59b42f7b2b41b8e6e1e5cb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:42:00.409502 18043 consensus_queue.cc:1035] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1882f6717380482186a4099e722dcbf2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:42:00.409797 18047 consensus_queue.cc:1035] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0ebd64af7440c5bd40b1b10a73cd92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:42:00.421267 18050 mvcc.cc:205] Tried to move back new op lower bound from 7334235833983795200 to 7334235833855434752. Current Snapshot: MvccSnapshot[applied={T|T < 7334235833983795200}]
I20260928 08:42:00.424733 18051 mvcc.cc:205] Tried to move back new op lower bound from 7334235833983795200 to 7334235833855434752. Current Snapshot: MvccSnapshot[applied={T|T < 7334235833983795200}]
I20260928 08:42:00.435496 17563 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:42:00.435791 17563 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:42:00.439319 17563 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95db7110595e4a318dce45a9182cd8f4: Sending DeleteTablet for 3 replicas of tablet 170995d6a7ed4553b4bbdf89c784956c
I20260928 08:42:00.440642 17954 tablet_service.cc:1514] Processing DeleteTablet for tablet 170995d6a7ed4553b4bbdf89c784956c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:42:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47926
I20260928 08:42:00.440780 17687 tablet_service.cc:1514] Processing DeleteTablet for tablet 170995d6a7ed4553b4bbdf89c784956c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:42:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39960
I20260928 08:42:00.441974 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 17624
I20260928 08:42:00.441959 17821 tablet_service.cc:1514] Processing DeleteTablet for tablet 170995d6a7ed4553b4bbdf89c784956c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:42:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39798
I20260928 08:42:00.464722 18061 tablet_replica.cc:330] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92: stopping tablet replica
I20260928 08:42:00.465408 18061 raft_consensus.cc:2229] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:42:00.466114 18061 raft_consensus.cc:2258] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:42:00.468349 18061 ts_tablet_manager.cc:1905] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 08:42:00 dist-test-slave-w6c0 krb5kdc[17518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:42:00.469893 18060 tablet_replica.cc:330] T 170995d6a7ed4553b4bbdf89c784956c P 1882f6717380482186a4099e722dcbf2: stopping tablet replica
I20260928 08:42:00.470438 18060 raft_consensus.cc:2229] T 170995d6a7ed4553b4bbdf89c784956c P 1882f6717380482186a4099e722dcbf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:42:00.469316 18059 tablet_replica.cc:330] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5: stopping tablet replica
I20260928 08:42:00.471088 18059 raft_consensus.cc:2229] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:42:00.471849 18059 raft_consensus.cc:2258] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:42:00.472446 18060 raft_consensus.cc:2258] T 170995d6a7ed4553b4bbdf89c784956c P 1882f6717380482186a4099e722dcbf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:42:00.474967 18060 ts_tablet_manager.cc:1905] T 170995d6a7ed4553b4bbdf89c784956c P 1882f6717380482186a4099e722dcbf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:42:00.477097 18061 ts_tablet_manager.cc:1918] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:42:00.477267 18061 log.cc:1196] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/170995d6a7ed4553b4bbdf89c784956c
I20260928 08:42:00.477795 18061 ts_tablet_manager.cc:1939] T 170995d6a7ed4553b4bbdf89c784956c P 8e0ebd64af7440c5bd40b1b10a73cd92: Deleting consensus metadata
I20260928 08:42:00.478200 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.442955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39968 (local address 127.0.97.1:44271)
0928 08:42:00.443184 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:00.443194 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:00.457055 (+ 13861us) server_negotiation.cc:407] Connection header received
0928 08:42:00.457587 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:00.457594 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:00.457732 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:00.457823 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:00.460038 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.461487 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.463905 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.464656 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.464768 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:00.470529 (+ 5761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:00.470547 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:00.470572 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:00.470616 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:00.474624 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:00.475560 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:00.475570 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:00.475573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:00.475721 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:00.476180 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:00.476191 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:00.476194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:00.476691 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:00.476993 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:00.477614 (+ 621us) server_negotiation.cc:299] Negotiation successful
0928 08:42:00.477841 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 08:42:00.478901 18059 ts_tablet_manager.cc:1905] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:42:00.479629 17747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:00.479765 17547 catalog_manager.cc:4841] TS 8e0ebd64af7440c5bd40b1b10a73cd92 (127.0.97.2:43731): tablet 170995d6a7ed4553b4bbdf89c784956c (table test-table [id=e1720f15b55843c6b36e6ed6acfc00cf]) successfully deleted
I20260928 08:42:00.487783 18059 ts_tablet_manager.cc:1918] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:42:00.488016 18059 log.cc:1196] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/170995d6a7ed4553b4bbdf89c784956c
I20260928 08:42:00.488611 18059 ts_tablet_manager.cc:1939] T 170995d6a7ed4553b4bbdf89c784956c P d48b7b5ba59b42f7b2b41b8e6e1e5cb5: Deleting consensus metadata
I20260928 08:42:00.490587 17548 catalog_manager.cc:4841] TS d48b7b5ba59b42f7b2b41b8e6e1e5cb5 (127.0.97.3:43995): tablet 170995d6a7ed4553b4bbdf89c784956c (table test-table [id=e1720f15b55843c6b36e6ed6acfc00cf]) successfully deleted
I20260928 08:42:00.570670 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 17624
W20260928 08:42:00.588526 17550 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44271 (error 108)
I20260928 08:42:00.589903 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 17758
I20260928 08:42:00.590416 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.589934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.0.1:39970)
0928 08:42:00.590120 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:00.590227 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 08:42:00.590775 17550 catalog_manager.cc:4584] TS 1882f6717380482186a4099e722dcbf2 (127.0.97.1:44271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 170995d6a7ed4553b4bbdf89c784956c: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44271: connect: Connection refused (error 111)
Sep 28 08:42:00 dist-test-slave-w6c0 krb5kdc[17518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:42:00.605795 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.590837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39808 (local address 127.0.97.2:43731)
0928 08:42:00.591064 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:00.591071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:00.591307 (+ 236us) server_negotiation.cc:407] Connection header received
0928 08:42:00.591438 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:00.591444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:00.591525 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:00.591636 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:00.592901 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.593635 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.595032 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.595412 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.595549 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:00.599872 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:00.599886 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:00.599901 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:00.599939 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:00.603015 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:00.603722 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:00.603728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:00.603731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:00.603854 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:00.604245 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:00.604249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:00.604251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:00.604640 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:00.604935 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:00.605252 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 08:42:00.605468 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:42:00.606479 17881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:00.644702 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.644142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.0.1:39980)
0928 08:42:00.644381 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:00.644500 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 08:42:00.645866 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.645462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.0.1:39984)
0928 08:42:00.645632 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:00.645727 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 08:42:00.703733 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 17758
I20260928 08:42:00.707257 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.706611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.0.1:39998)
0928 08:42:00.706943 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:00.707066 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260928 08:42:00.708460 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.707998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.0.1:40006)
0928 08:42:00.708215 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:00.708297 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 08:42:00.721401 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 17891
Sep 28 08:42:00 dist-test-slave-w6c0 krb5kdc[17518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:42:00.739496 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.722213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47950 (local address 127.0.97.3:43995)
0928 08:42:00.722385 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:00.722389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:00.722407 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:42:00.722611 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:00.722615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:00.722687 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:00.722776 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:00.724544 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.725367 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.727361 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:00.727805 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:00.727899 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:00.733340 (+ 5441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:00.733353 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:00.733369 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:00.733410 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:00.736623 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:00.737317 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:00.737329 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:00.737331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:00.737495 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:00.737896 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:00.737903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:00.737905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:00.738251 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:00.738601 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:00.738888 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 08:42:00.739112 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 08:42:00.740191 18014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:00.789075 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.788490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.0.1:40018)
0928 08:42:00.788740 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:00.788857 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 08:42:00.790269 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.789778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.0.1:40024)
0928 08:42:00.790034 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:00.790136 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260928 08:42:00.864992 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 17891
I20260928 08:42:00.884681 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 17531
I20260928 08:42:00.885836 17593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:00.998637 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 17531
I20260928 08:42:00.998772 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:00.998049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44271 (local address 127.0.0.1:40040)
0928 08:42:00.998384 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:00.998567 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
2026-09-28T08:42:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4258 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T08:42:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:01Z Disabled control of system clock
I20260928 08:42:01.075372 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33999
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:42:01.400986 18071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:01.401263 18071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:01.401422 18071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:01.408589 18071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:01.408684 18071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:01.408773 18071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:01.408883 18071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:01.408962 18071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:01.418073 18071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33999
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:01.419453 18071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:01.421211 18071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:01.432965 18077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:01.433058 18076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:01.432911 18079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:01.434355 18071 server_base.cc:1034] running on GCE node
I20260928 08:42:01.435129 18071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:01.436527 18071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:01.437754 18071 hybrid_clock.cc:648] HybridClock initialized: now 1790584921437706 us; error 36 us; skew 500 ppm
I20260928 08:42:01.438107 18071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:42:01.440299 18071 webserver.cc:460] Webserver started at http://127.0.97.62:41371/ using document root <none> and password file <none>
I20260928 08:42:01.440989 18071 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:01.441128 18071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:01.441442 18071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:01.444276 18071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "ef781e03978b4cf6a412f91e321efe34"
format_stamp: "Formatted at 2026-09-28 08:42:01 on dist-test-slave-w6c0"
server_key: "531de7d366fc7810f4f079c1b52a1ecc"
server_key_iv: "a42725de9271dcdc010c0f932f8816cb"
server_key_version: "encryptionkey@0"
I20260928 08:42:01.445032 18071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "ef781e03978b4cf6a412f91e321efe34"
format_stamp: "Formatted at 2026-09-28 08:42:01 on dist-test-slave-w6c0"
server_key: "531de7d366fc7810f4f079c1b52a1ecc"
server_key_iv: "a42725de9271dcdc010c0f932f8816cb"
server_key_version: "encryptionkey@0"
I20260928 08:42:01.449995 18071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:42:01.453227 18085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:01.454442 18071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 08:42:01.454638 18071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "ef781e03978b4cf6a412f91e321efe34"
format_stamp: "Formatted at 2026-09-28 08:42:01 on dist-test-slave-w6c0"
server_key: "531de7d366fc7810f4f079c1b52a1ecc"
server_key_iv: "a42725de9271dcdc010c0f932f8816cb"
server_key_version: "encryptionkey@0"
I20260928 08:42:01.454862 18071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:01.465865 18071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:01.466657 18071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:01.466923 18071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:01.484540 18071 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33999
I20260928 08:42:01.484558 18136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33999 every 8 connection(s)
I20260928 08:42:01.485834 18071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:01.487234 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 18071
I20260928 08:42:01.487426 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:01.487990 388 external_mini_cluster.cc:1428] Setting key 7937cdf94cd6523adeda53eb9f0034e6
I20260928 08:42:01.490360 18137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:01.500662 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:01.489826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34374 (local address 127.0.97.62:33999)
0928 08:42:01.490845 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:01.490856 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:01.490895 (+ 39us) server_negotiation.cc:407] Connection header received
0928 08:42:01.492488 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:01.492507 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:01.492915 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:01.493294 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:01.493829 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:01.495200 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:01.496514 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:01.497152 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:01.497349 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:01.498540 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:01.498553 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:01.498572 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:01.499131 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:01.499202 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:01.499222 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:01.499365 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:01.499522 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 08:42:01.499812 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":345,"threads_started":1}
I20260928 08:42:01.500818 18137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34: Bootstrap starting.
I20260928 08:42:01.504288 18137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:01.505173 18137 log.cc:824] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:01.507961 18137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34: No bootstrap required, opened a new log
I20260928 08:42:01.515276 18137 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef781e03978b4cf6a412f91e321efe34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33999 } }
I20260928 08:42:01.515578 18137 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:01.515645 18137 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef781e03978b4cf6a412f91e321efe34, State: Initialized, Role: FOLLOWER
I20260928 08:42:01.516206 18137 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef781e03978b4cf6a412f91e321efe34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33999 } }
I20260928 08:42:01.516523 18137 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:01.516665 18137 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:01.516875 18137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:01.518992 18137 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef781e03978b4cf6a412f91e321efe34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33999 } }
I20260928 08:42:01.519404 18137 leader_election.cc:302] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef781e03978b4cf6a412f91e321efe34; no voters:
I20260928 08:42:01.519879 18137 leader_election.cc:288] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:01.520021 18142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:01.520757 18142 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34 [term 1 LEADER]: Becoming Leader. State: Replica: ef781e03978b4cf6a412f91e321efe34, State: Running, Role: LEADER
I20260928 08:42:01.521333 18142 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef781e03978b4cf6a412f91e321efe34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33999 } }
I20260928 08:42:01.521967 18137 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:01.524751 18143 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef781e03978b4cf6a412f91e321efe34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef781e03978b4cf6a412f91e321efe34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33999 } } }
I20260928 08:42:01.525292 18143 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:01.525595 18144 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef781e03978b4cf6a412f91e321efe34. Latest consensus state: current_term: 1 leader_uuid: "ef781e03978b4cf6a412f91e321efe34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef781e03978b4cf6a412f91e321efe34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33999 } } }
I20260928 08:42:01.525934 18144 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:01.527161 18151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:01.533057 18151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:01.541540 18151 catalog_manager.cc:1347] Generated new cluster ID: 4ebed677747e4d9f998a7f22ab16e8fa
I20260928 08:42:01.541643 18151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:01.556442 18151 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:01.557700 18151 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:01.578179 18151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34: Generated new TSK 0
I20260928 08:42:01.579284 18151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:01.587646 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33999
--builtin_ntp_servers=127.0.97.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:42:01.909410 18161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:01.909755 18161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:01.909914 18161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:01.918081 18161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:01.918227 18161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:01.918411 18161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:42:01.927767 18161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:01.929051 18161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:01.930847 18161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:01.943547 18169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:01.943780 18166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:01.944002 18167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:01.944391 18161 server_base.cc:1034] running on GCE node
I20260928 08:42:01.944952 18161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:01.945863 18161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:01.947084 18161 hybrid_clock.cc:648] HybridClock initialized: now 1790584921947048 us; error 33 us; skew 500 ppm
I20260928 08:42:01.947394 18161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:42:01.949651 18161 webserver.cc:460] Webserver started at http://127.0.97.1:39007/ using document root <none> and password file <none>
I20260928 08:42:01.950410 18161 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:01.950577 18161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:01.950980 18161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:01.955235 18161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "57c4543ac22141e890dd0f8902c230f0"
format_stamp: "Formatted at 2026-09-28 08:42:01 on dist-test-slave-w6c0"
server_key: "e06d2d98669e2ed12c92250c54f981e8"
server_key_iv: "8a23f2084b105e04fa5684ed13967aeb"
server_key_version: "encryptionkey@0"
I20260928 08:42:01.956383 18161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "57c4543ac22141e890dd0f8902c230f0"
format_stamp: "Formatted at 2026-09-28 08:42:01 on dist-test-slave-w6c0"
server_key: "e06d2d98669e2ed12c92250c54f981e8"
server_key_iv: "8a23f2084b105e04fa5684ed13967aeb"
server_key_version: "encryptionkey@0"
I20260928 08:42:01.963279 18161 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 08:42:01.966939 18175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:01.968204 18161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:42:01.968355 18161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "57c4543ac22141e890dd0f8902c230f0"
format_stamp: "Formatted at 2026-09-28 08:42:01 on dist-test-slave-w6c0"
server_key: "e06d2d98669e2ed12c92250c54f981e8"
server_key_iv: "8a23f2084b105e04fa5684ed13967aeb"
server_key_version: "encryptionkey@0"
I20260928 08:42:01.968569 18161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:01.979125 18161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:01.979835 18161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:01.980093 18161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:01.980991 18161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:42:01.982363 18161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:42:01.982473 18161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:01.982626 18161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:42:01.982724 18161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:02.014977 18161 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38601
I20260928 08:42:02.015053 18287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38601 every 8 connection(s)
I20260928 08:42:02.016335 18161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:42:02.020466 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 18161
I20260928 08:42:02.020642 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:42:02.021196 388 external_mini_cluster.cc:1428] Setting key ca4707b24cb404fb06b80f267ed3abc2
I20260928 08:42:02.024160 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33999
--builtin_ntp_servers=127.0.97.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 08:42:02.030885 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:02.018864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40355 (local address 127.0.97.62:33999)
0928 08:42:02.019341 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:02.019348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:02.020686 (+ 1338us) server_negotiation.cc:407] Connection header received
0928 08:42:02.023059 (+ 2373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:02.023064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:02.023193 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:02.023296 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:02.024375 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.025596 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.027662 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.028428 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.028574 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:02.029874 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:02.029883 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:02.029890 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:02.029915 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:02.029977 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:02.029994 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:02.030098 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:02.030294 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 08:42:02.030405 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":227,"threads_started":1}
I20260928 08:42:02.031692 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:02.019498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33999 (local address 127.0.97.1:40355)
0928 08:42:02.020494 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:02.020550 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:02.022760 (+ 2210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:02.023531 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:02.023539 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:02.023636 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:02.024170 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:02.024185 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.025816 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.025825 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:02.027524 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:02.027536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.028639 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.028644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:02.028837 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:02.029556 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:42:02.029558 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:02.029595 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:42:02.029675 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:02.030092 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:02.030099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:02.030111 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 08:42:02.030248 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 08:42:02.030589 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":266,"threads_started":1}
I20260928 08:42:02.033401 18288 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33999
I20260928 08:42:02.033762 18288 heartbeater.cc:464] Registering TS with master...
I20260928 08:42:02.034647 18288 heartbeater.cc:511] Master 127.0.97.62:33999 requested a full tablet report, sending...
I20260928 08:42:02.036972 18102 ts_manager.cc:194] Registered new tserver with Master: 57c4543ac22141e890dd0f8902c230f0 (127.0.97.1:38601)
I20260928 08:42:02.038429 18102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:40355
W20260928 08:42:02.345943 18292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:02.346222 18292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:02.346374 18292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:02.353520 18292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:02.353613 18292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:02.353799 18292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:42:02.363247 18292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:02.364552 18292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:02.366266 18292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:02.379455 18297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:02.379381 18298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:02.379431 18300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:02.380237 18292 server_base.cc:1034] running on GCE node
I20260928 08:42:02.380745 18292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:02.381594 18292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:02.382793 18292 hybrid_clock.cc:648] HybridClock initialized: now 1790584922382763 us; error 29 us; skew 500 ppm
I20260928 08:42:02.383124 18292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:42:02.385277 18292 webserver.cc:460] Webserver started at http://127.0.97.2:41517/ using document root <none> and password file <none>
I20260928 08:42:02.385946 18292 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:02.386032 18292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:02.386356 18292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:02.389165 18292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "f62bcbc2faeb44879ee67acec798bfae"
format_stamp: "Formatted at 2026-09-28 08:42:02 on dist-test-slave-w6c0"
server_key: "a04db6ff14350361e035ab8d240d6b0b"
server_key_iv: "1c47a1b104e5235bbd5b1781bb9e1c0b"
server_key_version: "encryptionkey@0"
I20260928 08:42:02.390041 18292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f62bcbc2faeb44879ee67acec798bfae"
format_stamp: "Formatted at 2026-09-28 08:42:02 on dist-test-slave-w6c0"
server_key: "a04db6ff14350361e035ab8d240d6b0b"
server_key_iv: "1c47a1b104e5235bbd5b1781bb9e1c0b"
server_key_version: "encryptionkey@0"
I20260928 08:42:02.394984 18292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:42:02.398223 18306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:02.399521 18292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:42:02.399693 18292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "f62bcbc2faeb44879ee67acec798bfae"
format_stamp: "Formatted at 2026-09-28 08:42:02 on dist-test-slave-w6c0"
server_key: "a04db6ff14350361e035ab8d240d6b0b"
server_key_iv: "1c47a1b104e5235bbd5b1781bb9e1c0b"
server_key_version: "encryptionkey@0"
I20260928 08:42:02.399919 18292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:02.420042 18292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:02.420851 18292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:02.421139 18292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:02.422122 18292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:42:02.423866 18292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:42:02.423995 18292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:02.424132 18292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:42:02.424221 18292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:02.458847 18292 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36465
I20260928 08:42:02.458884 18418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36465 every 8 connection(s)
I20260928 08:42:02.460177 18292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:42:02.461313 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 18292
I20260928 08:42:02.461495 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:42:02.461990 388 external_mini_cluster.cc:1428] Setting key 8a679cd53e1f294bca1f81a70e274121
I20260928 08:42:02.464843 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33999
--builtin_ntp_servers=127.0.97.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 08:42:02.477072 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:02.463581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33999 (local address 127.0.97.2:60169)
0928 08:42:02.464667 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:02.464709 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:02.466992 (+ 2283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:02.467714 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:02.467722 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:02.467817 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:02.468361 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:02.468377 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.469837 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.469841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:02.471459 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:02.471491 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.472908 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.472917 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:02.473240 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:02.474220 (+ 980us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:42:02.474225 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:02.474276 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:42:02.474426 (+ 150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:02.474989 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:02.475002 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:02.475010 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 08:42:02.475228 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 08:42:02.475670 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":376,"threads_started":1}
I20260928 08:42:02.478528 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:02.462961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60169 (local address 127.0.97.62:33999)
0928 08:42:02.463137 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:02.463141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:02.464837 (+ 1696us) server_negotiation.cc:407] Connection header received
0928 08:42:02.467265 (+ 2428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:02.467273 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:02.467379 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:02.467499 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:02.468534 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.469678 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.471655 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.472392 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.472533 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:02.474758 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:02.474767 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:02.474771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:02.474793 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:02.474850 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:02.474867 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:02.474977 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:02.478015 (+ 3038us) server_negotiation.cc:299] Negotiation successful
0928 08:42:02.478137 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:42:02.479970 18419 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33999
I20260928 08:42:02.480288 18419 heartbeater.cc:464] Registering TS with master...
I20260928 08:42:02.481096 18419 heartbeater.cc:511] Master 127.0.97.62:33999 requested a full tablet report, sending...
I20260928 08:42:02.482769 18102 ts_manager.cc:194] Registered new tserver with Master: f62bcbc2faeb44879ee67acec798bfae (127.0.97.2:36465)
I20260928 08:42:02.483543 18102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:60169
W20260928 08:42:02.792270 18422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:02.792616 18422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:02.792776 18422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:02.800019 18422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:02.800161 18422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:02.800339 18422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:42:02.809592 18422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:02.811015 18422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:02.812798 18422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:02.825789 18427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:02.825860 18428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:02.825881 18430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:02.827049 18422 server_base.cc:1034] running on GCE node
I20260928 08:42:02.827736 18422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:02.828754 18422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:02.830015 18422 hybrid_clock.cc:648] HybridClock initialized: now 1790584922829972 us; error 33 us; skew 500 ppm
I20260928 08:42:02.830350 18422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:42:02.832685 18422 webserver.cc:460] Webserver started at http://127.0.97.3:36741/ using document root <none> and password file <none>
I20260928 08:42:02.833421 18422 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:02.833557 18422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:02.833882 18422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:02.836791 18422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "eeb6b389e9a64a1f8e3592a122e4928b"
format_stamp: "Formatted at 2026-09-28 08:42:02 on dist-test-slave-w6c0"
server_key: "887bf3c4900c03c8cb361318f2a5e432"
server_key_iv: "e043917d35c5e472fa32418820c9cd65"
server_key_version: "encryptionkey@0"
I20260928 08:42:02.837603 18422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "eeb6b389e9a64a1f8e3592a122e4928b"
format_stamp: "Formatted at 2026-09-28 08:42:02 on dist-test-slave-w6c0"
server_key: "887bf3c4900c03c8cb361318f2a5e432"
server_key_iv: "e043917d35c5e472fa32418820c9cd65"
server_key_version: "encryptionkey@0"
I20260928 08:42:02.842751 18422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:42:02.846030 18436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:02.847357 18422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:42:02.847535 18422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "eeb6b389e9a64a1f8e3592a122e4928b"
format_stamp: "Formatted at 2026-09-28 08:42:02 on dist-test-slave-w6c0"
server_key: "887bf3c4900c03c8cb361318f2a5e432"
server_key_iv: "e043917d35c5e472fa32418820c9cd65"
server_key_version: "encryptionkey@0"
I20260928 08:42:02.847755 18422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:02.863734 18422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:02.864491 18422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:02.864763 18422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:02.865659 18422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:42:02.867087 18422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:42:02.867218 18422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:02.867354 18422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:42:02.867444 18422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:02.900377 18422 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34089
I20260928 08:42:02.900463 18548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34089 every 8 connection(s)
I20260928 08:42:02.901734 18422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:42:02.909567 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 18422
I20260928 08:42:02.909747 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:42:02.910181 388 external_mini_cluster.cc:1428] Setting key a251d9eeba2629e2e11c3932d88fce18
I20260928 08:42:02.914295 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:02.904367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46083 (local address 127.0.97.62:33999)
0928 08:42:02.904555 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:02.904559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:02.905837 (+ 1278us) server_negotiation.cc:407] Connection header received
0928 08:42:02.907694 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:02.907700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:02.907801 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:02.907911 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:02.908963 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.909750 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.911525 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.911944 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.912083 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:02.913439 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:02.913445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:02.913457 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:02.913475 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:02.913505 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:02.913517 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:02.913610 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:02.913838 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 08:42:02.913971 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:42:02.915261 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:02.904831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33999 (local address 127.0.97.3:46083)
0928 08:42:02.905681 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:02.905720 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:02.907411 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:02.908140 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:02.908148 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:02.908237 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:02.908779 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:02.908795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.909908 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.909914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:02.911383 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:02.911395 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.912164 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.912170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:02.912401 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:02.913118 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:42:02.913121 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:02.913158 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:42:02.913244 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:02.913611 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:02.913624 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:02.913634 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 08:42:02.913810 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 08:42:02.914180 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":271,"threads_started":1}
I20260928 08:42:02.916471 18549 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33999
I20260928 08:42:02.916751 18549 heartbeater.cc:464] Registering TS with master...
I20260928 08:42:02.917471 18549 heartbeater.cc:511] Master 127.0.97.62:33999 requested a full tablet report, sending...
I20260928 08:42:02.918848 18102 ts_manager.cc:194] Registered new tserver with Master: eeb6b389e9a64a1f8e3592a122e4928b (127.0.97.3:34089)
I20260928 08:42:02.919633 18102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:46083
I20260928 08:42:02.924919 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:42:02.934254 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:02.928917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34376 (local address 127.0.97.62:33999)
0928 08:42:02.929084 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:02.929089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:02.929509 (+ 420us) server_negotiation.cc:407] Connection header received
0928 08:42:02.929644 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:02.929648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:02.929713 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:02.929804 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:02.930226 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.930913 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.932279 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.932671 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.932758 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:02.933719 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:02.933723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:02.933726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:02.933742 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:02.933768 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:02.933779 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:02.933854 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:02.933973 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 08:42:02.934039 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 08:42:02.937755 18102 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:42:02.944731 18102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:42:02.979907 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:02.956979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38601 (local address 127.0.0.1:51880)
0928 08:42:02.969152 (+ 12173us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:02.969213 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:02.969425 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:02.970904 (+ 1479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:02.970916 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:02.970965 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:02.971495 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:02.971512 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.974047 (+ 2535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.974052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:02.976608 (+ 2556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:02.976622 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.978479 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.978489 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:02.979036 (+ 547us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:02.979097 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 08:42:02.979237 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 08:42:02.979432 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11860,"mutex_wait_us":74,"thread_start_us":16847,"threads_started":1}
I20260928 08:42:02.981001 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:02.957131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.0.97.1:38601)
0928 08:42:02.970116 (+ 12985us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:02.970129 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:02.970166 (+ 37us) server_negotiation.cc:407] Connection header received
0928 08:42:02.970279 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:02.970290 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:02.970631 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:02.970861 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:02.971878 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.973870 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.976808 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.978270 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.978409 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:02.978691 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:02.980471 (+ 1780us) server_negotiation.cc:299] Negotiation successful
0928 08:42:02.980662 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12777,"thread_start_us":12658,"threads_started":1}
I20260928 08:42:02.982829 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:02.957242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34089 (local address 127.0.0.1:52138)
0928 08:42:02.973932 (+ 16690us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:02.973955 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:02.974085 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:02.974730 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:02.974734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:02.974761 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:02.976218 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:02.976229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.978085 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.978090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:02.980466 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:02.980479 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.982088 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.982093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:02.982241 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:02.982292 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 08:42:02.982355 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 08:42:02.982451 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16514,"mutex_wait_us":62,"thread_start_us":13874,"threads_started":1}
I20260928 08:42:02.982944 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:02.957577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52138 (local address 127.0.97.3:34089)
0928 08:42:02.973447 (+ 15870us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:02.973454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:02.974051 (+ 597us) server_negotiation.cc:407] Connection header received
0928 08:42:02.974286 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:02.974291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:02.974559 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:02.974734 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:02.976415 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.977893 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.980655 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.981892 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.982032 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:02.982311 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:02.982392 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 08:42:02.982582 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15671,"thread_start_us":15497,"threads_started":1}
I20260928 08:42:02.983161 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:02.959366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52426 (local address 127.0.97.2:36465)
0928 08:42:02.973683 (+ 14317us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:02.973691 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:02.974244 (+ 553us) server_negotiation.cc:407] Connection header received
0928 08:42:02.974609 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:02.974620 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:02.974832 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:02.974988 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:02.976041 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.977366 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.980347 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.981553 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.981659 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:02.981824 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:02.982639 (+ 815us) server_negotiation.cc:299] Negotiation successful
0928 08:42:02.982834 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14153,"thread_start_us":14023,"threads_started":1}
I20260928 08:42:02.983650 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:02.958618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36465 (local address 127.0.0.1:52426)
0928 08:42:02.974129 (+ 15511us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:02.974151 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:02.974372 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:02.975353 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:02.975358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:02.975383 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:02.975846 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:02.975857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.977499 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.977506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:02.980124 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:02.980138 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:02.981958 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:02.981991 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:02.982185 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:02.982218 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 08:42:02.982304 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 08:42:02.982438 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15374,"mutex_wait_us":1,"thread_start_us":15008,"threads_started":1}
I20260928 08:42:02.985280 18223 tablet_service.cc:1467] Processing CreateTablet for tablet 73569a51972d4db49a7635082c4affc8 (DEFAULT_TABLE table=test-table [id=0d52a1c9c5eb4391be6caedcdc586280]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:42:02.986817 18223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73569a51972d4db49a7635082c4affc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:02.986550 18484 tablet_service.cc:1467] Processing CreateTablet for tablet 73569a51972d4db49a7635082c4affc8 (DEFAULT_TABLE table=test-table [id=0d52a1c9c5eb4391be6caedcdc586280]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:42:02.986711 18354 tablet_service.cc:1467] Processing CreateTablet for tablet 73569a51972d4db49a7635082c4affc8 (DEFAULT_TABLE table=test-table [id=0d52a1c9c5eb4391be6caedcdc586280]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:42:02.987795 18354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73569a51972d4db49a7635082c4affc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:02.987926 18484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73569a51972d4db49a7635082c4affc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:02.998421 18567 tablet_bootstrap.cc:492] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae: Bootstrap starting.
I20260928 08:42:03.001636 18568 tablet_bootstrap.cc:492] T 73569a51972d4db49a7635082c4affc8 P 57c4543ac22141e890dd0f8902c230f0: Bootstrap starting.
I20260928 08:42:03.003279 18567 tablet_bootstrap.cc:654] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:03.003400 18569 tablet_bootstrap.cc:492] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b: Bootstrap starting.
I20260928 08:42:03.004755 18567 log.cc:824] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:03.005295 18568 tablet_bootstrap.cc:654] T 73569a51972d4db49a7635082c4affc8 P 57c4543ac22141e890dd0f8902c230f0: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:03.006366 18568 log.cc:824] T 73569a51972d4db49a7635082c4affc8 P 57c4543ac22141e890dd0f8902c230f0: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:03.006644 18569 tablet_bootstrap.cc:654] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:03.007817 18569 log.cc:824] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:03.007877 18567 tablet_bootstrap.cc:492] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae: No bootstrap required, opened a new log
I20260928 08:42:03.008136 18567 ts_tablet_manager.cc:1397] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260928 08:42:03.009274 18568 tablet_bootstrap.cc:492] T 73569a51972d4db49a7635082c4affc8 P 57c4543ac22141e890dd0f8902c230f0: No bootstrap required, opened a new log
I20260928 08:42:03.009536 18568 ts_tablet_manager.cc:1397] T 73569a51972d4db49a7635082c4affc8 P 57c4543ac22141e890dd0f8902c230f0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 08:42:03.019073 18569 tablet_bootstrap.cc:492] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b: No bootstrap required, opened a new log
I20260928 08:42:03.018774 18567 raft_consensus.cc:348] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c4543ac22141e890dd0f8902c230f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38601 } } peers { permanent_uuid: "eeb6b389e9a64a1f8e3592a122e4928b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34089 } } peers { permanent_uuid: "f62bcbc2faeb44879ee67acec798bfae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36465 } }
I20260928 08:42:03.019218 18567 raft_consensus.cc:374] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:03.019337 18567 raft_consensus.cc:729] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f62bcbc2faeb44879ee67acec798bfae, State: Initialized, Role: FOLLOWER
I20260928 08:42:03.019402 18569 ts_tablet_manager.cc:1397] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.013s
I20260928 08:42:03.019995 18567 consensus_queue.cc:260] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c4543ac22141e890dd0f8902c230f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38601 } } peers { permanent_uuid: "eeb6b389e9a64a1f8e3592a122e4928b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34089 } } peers { permanent_uuid: "f62bcbc2faeb44879ee67acec798bfae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36465 } }
I20260928 08:42:03.021732 18567 ts_tablet_manager.cc:1428] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae: Time spent starting tablet: real 0.013s user 0.007s sys 0.007s
I20260928 08:42:03.022228 18419 heartbeater.cc:503] Master 127.0.97.62:33999 was elected leader, sending a full tablet report...
I20260928 08:42:03.022781 18568 raft_consensus.cc:348] T 73569a51972d4db49a7635082c4affc8 P 57c4543ac22141e890dd0f8902c230f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c4543ac22141e890dd0f8902c230f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38601 } } peers { permanent_uuid: "eeb6b389e9a64a1f8e3592a122e4928b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34089 } } peers { permanent_uuid: "f62bcbc2faeb44879ee67acec798bfae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36465 } }
I20260928 08:42:03.023275 18568 raft_consensus.cc:374] T 73569a51972d4db49a7635082c4affc8 P 57c4543ac22141e890dd0f8902c230f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:03.023371 18568 raft_consensus.cc:729] T 73569a51972d4db49a7635082c4affc8 P 57c4543ac22141e890dd0f8902c230f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57c4543ac22141e890dd0f8902c230f0, State: Initialized, Role: FOLLOWER
I20260928 08:42:03.024042 18568 consensus_queue.cc:260] T 73569a51972d4db49a7635082c4affc8 P 57c4543ac22141e890dd0f8902c230f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c4543ac22141e890dd0f8902c230f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38601 } } peers { permanent_uuid: "eeb6b389e9a64a1f8e3592a122e4928b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34089 } } peers { permanent_uuid: "f62bcbc2faeb44879ee67acec798bfae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36465 } }
I20260928 08:42:03.029656 18569 raft_consensus.cc:348] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c4543ac22141e890dd0f8902c230f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38601 } } peers { permanent_uuid: "eeb6b389e9a64a1f8e3592a122e4928b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34089 } } peers { permanent_uuid: "f62bcbc2faeb44879ee67acec798bfae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36465 } }
I20260928 08:42:03.030092 18569 raft_consensus.cc:374] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:03.030197 18569 raft_consensus.cc:729] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb6b389e9a64a1f8e3592a122e4928b, State: Initialized, Role: FOLLOWER
I20260928 08:42:03.030834 18569 consensus_queue.cc:260] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c4543ac22141e890dd0f8902c230f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38601 } } peers { permanent_uuid: "eeb6b389e9a64a1f8e3592a122e4928b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34089 } } peers { permanent_uuid: "f62bcbc2faeb44879ee67acec798bfae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36465 } }
I20260928 08:42:03.032032 18568 ts_tablet_manager.cc:1428] T 73569a51972d4db49a7635082c4affc8 P 57c4543ac22141e890dd0f8902c230f0: Time spent starting tablet: real 0.022s user 0.013s sys 0.008s
I20260928 08:42:03.032121 18288 heartbeater.cc:503] Master 127.0.97.62:33999 was elected leader, sending a full tablet report...
I20260928 08:42:03.032917 18549 heartbeater.cc:503] Master 127.0.97.62:33999 was elected leader, sending a full tablet report...
I20260928 08:42:03.033504 18569 ts_tablet_manager.cc:1428] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260928 08:42:03.096460 18575 raft_consensus.cc:482] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:42:03.096788 18575 raft_consensus.cc:504] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c4543ac22141e890dd0f8902c230f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38601 } } peers { permanent_uuid: "eeb6b389e9a64a1f8e3592a122e4928b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34089 } } peers { permanent_uuid: "f62bcbc2faeb44879ee67acec798bfae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36465 } }
I20260928 08:42:03.098495 18575 leader_election.cc:288] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57c4543ac22141e890dd0f8902c230f0 (127.0.97.1:38601), f62bcbc2faeb44879ee67acec798bfae (127.0.97.2:36465)
I20260928 08:42:03.104944 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:03.098927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52607 (local address 127.0.97.2:36465)
0928 08:42:03.099130 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:03.099135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:03.099154 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:42:03.099379 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:03.099387 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:03.099473 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:03.099567 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:03.100132 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:03.101077 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:03.102994 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:03.103939 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:03.104037 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:03.104104 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:03.104514 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 08:42:03.104665 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:42:03.104928 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:03.098764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36465 (local address 127.0.97.3:52607)
0928 08:42:03.099025 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:03.099046 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:03.099172 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:03.099618 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:03.099622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:03.099652 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:03.099921 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:03.099932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:03.101216 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:03.101221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:03.102823 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:03.102835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:03.104087 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:03.104094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:03.104308 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:03.104365 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 08:42:03.104464 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 08:42:03.104626 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260928 08:42:03.105437 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:03.098809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38601 (local address 127.0.97.3:50671)
0928 08:42:03.099649 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:03.099667 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:03.099790 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:03.100165 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:03.100172 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:03.100202 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:03.100570 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:03.100580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:03.101732 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:03.101737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:03.103263 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:03.103279 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:03.104655 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:03.104661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:03.104857 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:03.104889 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 08:42:03.104990 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 08:42:03.105124 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":492,"threads_started":1}
I20260928 08:42:03.105453 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:03.099094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50671 (local address 127.0.97.1:38601)
0928 08:42:03.099362 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:03.099368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:03.099772 (+ 404us) server_negotiation.cc:407] Connection header received
0928 08:42:03.099921 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:03.099926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:03.100028 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:03.100123 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:03.100722 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:03.101581 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:03.103445 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:03.104491 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:03.104603 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:03.104699 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:03.105043 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 08:42:03.105214 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 08:42:03.106084 18374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73569a51972d4db49a7635082c4affc8" candidate_uuid: "eeb6b389e9a64a1f8e3592a122e4928b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f62bcbc2faeb44879ee67acec798bfae" is_pre_election: true
I20260928 08:42:03.106657 18374 raft_consensus.cc:2454] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb6b389e9a64a1f8e3592a122e4928b in term 0.
I20260928 08:42:03.106647 18243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73569a51972d4db49a7635082c4affc8" candidate_uuid: "eeb6b389e9a64a1f8e3592a122e4928b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57c4543ac22141e890dd0f8902c230f0" is_pre_election: true
I20260928 08:42:03.107148 18243 raft_consensus.cc:2454] T 73569a51972d4db49a7635082c4affc8 P 57c4543ac22141e890dd0f8902c230f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb6b389e9a64a1f8e3592a122e4928b in term 0.
I20260928 08:42:03.107391 18440 leader_election.cc:302] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eeb6b389e9a64a1f8e3592a122e4928b, f62bcbc2faeb44879ee67acec798bfae; no voters:
I20260928 08:42:03.107813 18575 raft_consensus.cc:2781] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:42:03.107968 18575 raft_consensus.cc:482] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:42:03.108076 18575 raft_consensus.cc:3037] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:03.110225 18575 raft_consensus.cc:504] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c4543ac22141e890dd0f8902c230f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38601 } } peers { permanent_uuid: "eeb6b389e9a64a1f8e3592a122e4928b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34089 } } peers { permanent_uuid: "f62bcbc2faeb44879ee67acec798bfae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36465 } }
I20260928 08:42:03.110870 18575 leader_election.cc:288] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [CANDIDATE]: Term 1 election: Requested vote from peers 57c4543ac22141e890dd0f8902c230f0 (127.0.97.1:38601), f62bcbc2faeb44879ee67acec798bfae (127.0.97.2:36465)
I20260928 08:42:03.111267 18243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73569a51972d4db49a7635082c4affc8" candidate_uuid: "eeb6b389e9a64a1f8e3592a122e4928b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57c4543ac22141e890dd0f8902c230f0"
I20260928 08:42:03.111320 18374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73569a51972d4db49a7635082c4affc8" candidate_uuid: "eeb6b389e9a64a1f8e3592a122e4928b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f62bcbc2faeb44879ee67acec798bfae"
I20260928 08:42:03.111472 18243 raft_consensus.cc:3037] T 73569a51972d4db49a7635082c4affc8 P 57c4543ac22141e890dd0f8902c230f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:03.111512 18374 raft_consensus.cc:3037] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:03.113598 18243 raft_consensus.cc:2454] T 73569a51972d4db49a7635082c4affc8 P 57c4543ac22141e890dd0f8902c230f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb6b389e9a64a1f8e3592a122e4928b in term 1.
I20260928 08:42:03.113664 18374 raft_consensus.cc:2454] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb6b389e9a64a1f8e3592a122e4928b in term 1.
I20260928 08:42:03.113988 18438 leader_election.cc:302] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57c4543ac22141e890dd0f8902c230f0, eeb6b389e9a64a1f8e3592a122e4928b; no voters:
I20260928 08:42:03.114310 18575 raft_consensus.cc:2781] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:03.114795 18575 raft_consensus.cc:686] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [term 1 LEADER]: Becoming Leader. State: Replica: eeb6b389e9a64a1f8e3592a122e4928b, State: Running, Role: LEADER
I20260928 08:42:03.115260 18575 consensus_queue.cc:237] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c4543ac22141e890dd0f8902c230f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38601 } } peers { permanent_uuid: "eeb6b389e9a64a1f8e3592a122e4928b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34089 } } peers { permanent_uuid: "f62bcbc2faeb44879ee67acec798bfae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36465 } }
I20260928 08:42:03.120065 18100 catalog_manager.cc:5495] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b reported cstate change: term changed from 0 to 1, leader changed from <none> to eeb6b389e9a64a1f8e3592a122e4928b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "eeb6b389e9a64a1f8e3592a122e4928b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c4543ac22141e890dd0f8902c230f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeb6b389e9a64a1f8e3592a122e4928b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f62bcbc2faeb44879ee67acec798bfae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36465 } health_report { overall_health: UNKNOWN } } }
I20260928 08:42:03.137744 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:03.132338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.0.97.3:34089)
0928 08:42:03.132540 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:03.132546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:03.132564 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:42:03.132771 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:03.132775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:03.132861 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:03.133010 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:42:03.133361 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:03.134099 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:03.135574 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:03.135981 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:03.136122 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:03.136659 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:42:03.137152 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:42:03.137340 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:03.137425 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 08:42:03.137500 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:42:03.145507 18374 raft_consensus.cc:1261] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae [term 1 FOLLOWER]: Refusing update from remote peer eeb6b389e9a64a1f8e3592a122e4928b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:42:03.145625 18243 raft_consensus.cc:1261] T 73569a51972d4db49a7635082c4affc8 P 57c4543ac22141e890dd0f8902c230f0 [term 1 FOLLOWER]: Refusing update from remote peer eeb6b389e9a64a1f8e3592a122e4928b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:42:03.146590 18577 consensus_queue.cc:1035] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [LEADER]: Connected to new peer: Peer: permanent_uuid: "57c4543ac22141e890dd0f8902c230f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:42:03.146929 18575 consensus_queue.cc:1035] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f62bcbc2faeb44879ee67acec798bfae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 08:42:03.153414 18550 tablet.cc:2369] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:42:03.157801 18580 mvcc.cc:205] Tried to move back new op lower bound from 7334235845195390976 to 7334235845084626944. Current Snapshot: MvccSnapshot[applied={T|T < 7334235845195390976}]
I20260928 08:42:03.160044 18582 mvcc.cc:205] Tried to move back new op lower bound from 7334235845195390976 to 7334235845084626944. Current Snapshot: MvccSnapshot[applied={T|T < 7334235845195390976}]
I20260928 08:42:03.160388 18581 mvcc.cc:205] Tried to move back new op lower bound from 7334235845195390976 to 7334235845084626944. Current Snapshot: MvccSnapshot[applied={T|T < 7334235845195390976}]
I20260928 08:42:03.177665 18100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:42:03.177953 18100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:42:03.181483 18100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef781e03978b4cf6a412f91e321efe34: Sending DeleteTablet for 3 replicas of tablet 73569a51972d4db49a7635082c4affc8
I20260928 08:42:03.182596 18484 tablet_service.cc:1514] Processing DeleteTablet for tablet 73569a51972d4db49a7635082c4affc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:42:03 UTC) from {username='slave'} at 127.0.0.1:52138
I20260928 08:42:03.182633 18223 tablet_service.cc:1514] Processing DeleteTablet for tablet 73569a51972d4db49a7635082c4affc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:42:03 UTC) from {username='slave'} at 127.0.0.1:51880
I20260928 08:42:03.183462 18590 tablet_replica.cc:330] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b: stopping tablet replica
I20260928 08:42:03.183502 18354 tablet_service.cc:1514] Processing DeleteTablet for tablet 73569a51972d4db49a7635082c4affc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:42:03 UTC) from {username='slave'} at 127.0.0.1:52426
I20260928 08:42:03.183995 18590 raft_consensus.cc:2229] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:42:03.184453 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 18161
I20260928 08:42:03.184757 18590 raft_consensus.cc:2258] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:42:03.186919 18590 ts_tablet_manager.cc:1905] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:42:03.196254 18590 ts_tablet_manager.cc:1918] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:42:03.196507 18590 log.cc:1196] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/73569a51972d4db49a7635082c4affc8
I20260928 08:42:03.197192 18590 ts_tablet_manager.cc:1939] T 73569a51972d4db49a7635082c4affc8 P eeb6b389e9a64a1f8e3592a122e4928b: Deleting consensus metadata
I20260928 08:42:03.199126 18086 catalog_manager.cc:4841] TS eeb6b389e9a64a1f8e3592a122e4928b (127.0.97.3:34089): tablet 73569a51972d4db49a7635082c4affc8 (table test-table [id=0d52a1c9c5eb4391be6caedcdc586280]) successfully deleted
I20260928 08:42:03.204826 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:03.185893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.0.97.1:38601)
0928 08:42:03.186073 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:03.186080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:03.195089 (+ 9009us) server_negotiation.cc:407] Connection header received
0928 08:42:03.195303 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:03.195334 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:03.195464 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:03.195610 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:03.196541 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:03.198140 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:03.200738 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:03.201670 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:03.201818 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:03.203227 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:03.203240 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:03.203257 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:03.204029 (+ 772us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:03.204077 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:03.204097 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:03.204218 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:03.204331 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 08:42:03.204441 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:42:03.205618 18283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:03.211818 18592 tablet_replica.cc:330] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae: stopping tablet replica
I20260928 08:42:03.213122 18592 raft_consensus.cc:2229] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:42:03.213698 18592 raft_consensus.cc:2258] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:42:03.215960 18592 ts_tablet_manager.cc:1905] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:42:03.223137 18592 ts_tablet_manager.cc:1918] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:42:03.223299 18592 log.cc:1196] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/73569a51972d4db49a7635082c4affc8
I20260928 08:42:03.223850 18592 ts_tablet_manager.cc:1939] T 73569a51972d4db49a7635082c4affc8 P f62bcbc2faeb44879ee67acec798bfae: Deleting consensus metadata
I20260928 08:42:03.225772 18089 catalog_manager.cc:4841] TS f62bcbc2faeb44879ee67acec798bfae (127.0.97.2:36465): tablet 73569a51972d4db49a7635082c4affc8 (table test-table [id=0d52a1c9c5eb4391be6caedcdc586280]) successfully deleted
I20260928 08:42:03.300097 18591 tablet_replica.cc:330] T 73569a51972d4db49a7635082c4affc8 P 57c4543ac22141e890dd0f8902c230f0: stopping tablet replica
I20260928 08:42:03.300268 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 18161
I20260928 08:42:03.318310 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 18292
W20260928 08:42:03.318228 18087 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38601 (error 108)
I20260928 08:42:03.320468 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:03.320014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38601 (local address 127.0.0.1:51896)
0928 08:42:03.320203 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:03.320332 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 08:42:03.320734 18087 catalog_manager.cc:4584] TS 57c4543ac22141e890dd0f8902c230f0 (127.0.97.1:38601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73569a51972d4db49a7635082c4affc8: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38601: connect: Connection refused (error 111)
I20260928 08:42:03.327003 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:03.319649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.0.97.2:36465)
0928 08:42:03.319833 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:03.319838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:03.319854 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:42:03.319909 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:03.319913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:03.319993 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:03.320128 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:03.320718 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:03.321876 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:03.323522 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:03.324175 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:03.324306 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:03.325346 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:03.325359 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:03.325378 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:03.326159 (+ 781us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:03.326222 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:03.326249 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:03.326380 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:03.326493 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 08:42:03.326644 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:42:03.327817 18414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:03.374644 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:03.374040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38601 (local address 127.0.0.1:51900)
0928 08:42:03.374286 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:03.374402 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 08:42:03.375797 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:03.375385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38601 (local address 127.0.0.1:51916)
0928 08:42:03.375549 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:03.375656 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 08:42:03.419507 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 18292
I20260928 08:42:03.436239 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:03.435782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38601 (local address 127.0.0.1:51926)
0928 08:42:03.435988 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:03.436076 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 08:42:03.437325 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:03.436972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38601 (local address 127.0.0.1:51942)
0928 08:42:03.437135 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:03.437221 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 08:42:03.443089 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 18422
I20260928 08:42:03.450084 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:03.443850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52152 (local address 127.0.97.3:34089)
0928 08:42:03.444046 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:03.444052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:03.444073 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:42:03.444171 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:03.444176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:03.444278 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:03.444394 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:03.444820 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:03.445678 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:03.446980 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:03.447429 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:03.447575 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:03.448668 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:03.448677 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:03.448691 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:03.449295 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:03.449346 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:03.449364 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:03.449489 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:03.449635 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 08:42:03.449727 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 08:42:03.450870 18544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:03.518024 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:03.517506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38601 (local address 127.0.0.1:51948)
0928 08:42:03.517747 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:03.517847 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260928 08:42:03.519173 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:03.518801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38601 (local address 127.0.0.1:51964)
0928 08:42:03.518991 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:03.519057 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 08:42:03.572674 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 18422
I20260928 08:42:03.592154 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 18071
I20260928 08:42:03.593216 18132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:03.703536 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:03.702849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38601 (local address 127.0.0.1:51976)
0928 08:42:03.703085 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:03.703291 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 08:42:03.703606 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 18071
2026-09-28T08:42:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2703 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:03 dist-test-slave-w6c0 krb5kdc[18601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:03 dist-test-slave-w6c0 krb5kdc[18601](info): set up 2 sockets
Sep 28 08:42:03 dist-test-slave-w6c0 krb5kdc[18601](info): commencing operation
krb5kdc: starting...
W20260928 08:42:04.814668 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:04 dist-test-slave-w6c0 krb5kdc[18601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:42:05.058967 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41859
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:05.385823 18614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:05.386212 18614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:05.386382 18614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:05.393661 18614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:05.393766 18614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:05.393829 18614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:05.393945 18614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:05.394021 18614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:05.403931 18614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:05.405288 18614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:05.406595 18614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:05.418121 18619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:05.418207 18622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:05.418233 18620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:05.419154 18614 server_base.cc:1034] running on GCE node
I20260928 08:42:05.419822 18614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:05.421134 18614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:05.422339 18614 hybrid_clock.cc:648] HybridClock initialized: now 1790584925422299 us; error 33 us; skew 500 ppm
I20260928 08:42:05.422735 18614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:05 dist-test-slave-w6c0 krb5kdc[18601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:05.432006 18614 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:42:05.433621 18614 webserver.cc:460] Webserver started at http://127.0.97.62:41805/ using document root <none> and password file <none>
I20260928 08:42:05.434322 18614 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:05.434474 18614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:05.434826 18614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:05.437754 18614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "917b4b5757b5497c971a7efca3f35f86"
format_stamp: "Formatted at 2026-09-28 08:42:05 on dist-test-slave-w6c0"
server_key: "4acab2b02243754d461b4b004a47bf12"
server_key_iv: "335a30f7e1f8038cf9fb03e97d85c8c8"
server_key_version: "encryptionkey@0"
I20260928 08:42:05.438596 18614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "917b4b5757b5497c971a7efca3f35f86"
format_stamp: "Formatted at 2026-09-28 08:42:05 on dist-test-slave-w6c0"
server_key: "4acab2b02243754d461b4b004a47bf12"
server_key_iv: "335a30f7e1f8038cf9fb03e97d85c8c8"
server_key_version: "encryptionkey@0"
I20260928 08:42:05.443629 18614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:42:05.446971 18629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:05.448199 18614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 08:42:05.448402 18614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "917b4b5757b5497c971a7efca3f35f86"
format_stamp: "Formatted at 2026-09-28 08:42:05 on dist-test-slave-w6c0"
server_key: "4acab2b02243754d461b4b004a47bf12"
server_key_iv: "335a30f7e1f8038cf9fb03e97d85c8c8"
server_key_version: "encryptionkey@0"
I20260928 08:42:05.448613 18614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:05.466229 18614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:05.470118 18614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:05.470408 18614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:05.488831 18614 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41859
I20260928 08:42:05.488864 18680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41859 every 8 connection(s)
I20260928 08:42:05.490255 18614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:05.492101 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 18614
I20260928 08:42:05.492285 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:05.492849 388 external_mini_cluster.cc:1428] Setting key 60e0989a08695f676c31612a606d9538
I20260928 08:42:05.494843 18681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:05.506477 18681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86: Bootstrap starting.
Sep 28 08:42:05 dist-test-slave-w6c0 krb5kdc[18601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:42:05.510339 18681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:05.511515 18681 log.cc:824] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:05.515216 18681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86: No bootstrap required, opened a new log
I20260928 08:42:05.515758 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:05.494775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38666 (local address 127.0.97.62:41859)
0928 08:42:05.495513 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:05.495525 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:05.495564 (+ 39us) server_negotiation.cc:407] Connection header received
0928 08:42:05.497219 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:05.497239 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:05.497637 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:05.498037 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:05.499550 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:05.500880 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:05.502158 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:05.502942 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:05.503144 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:05.507871 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:05.507879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:05.507893 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:05.507936 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:05.511224 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:05.512196 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:05.512207 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:05.512218 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:05.512456 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:05.512843 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:05.512849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:05.512852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:05.513436 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:05.513650 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:05.514215 (+ 565us) server_negotiation.cc:299] Negotiation successful
0928 08:42:05.514725 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":331,"threads_started":1}
I20260928 08:42:05.523156 18681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917b4b5757b5497c971a7efca3f35f86" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41859 } }
I20260928 08:42:05.523586 18681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:05.523702 18681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 917b4b5757b5497c971a7efca3f35f86, State: Initialized, Role: FOLLOWER
I20260928 08:42:05.524286 18681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917b4b5757b5497c971a7efca3f35f86" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41859 } }
I20260928 08:42:05.524521 18681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:05.524659 18681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:05.524861 18681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:05.527163 18681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917b4b5757b5497c971a7efca3f35f86" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41859 } }
I20260928 08:42:05.527654 18681 leader_election.cc:302] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 917b4b5757b5497c971a7efca3f35f86; no voters:
I20260928 08:42:05.528245 18681 leader_election.cc:288] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:05.528373 18686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:05.529498 18686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86 [term 1 LEADER]: Becoming Leader. State: Replica: 917b4b5757b5497c971a7efca3f35f86, State: Running, Role: LEADER
I20260928 08:42:05.530129 18686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917b4b5757b5497c971a7efca3f35f86" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41859 } }
I20260928 08:42:05.530625 18681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:05.532974 18688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 917b4b5757b5497c971a7efca3f35f86. Latest consensus state: current_term: 1 leader_uuid: "917b4b5757b5497c971a7efca3f35f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917b4b5757b5497c971a7efca3f35f86" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41859 } } }
I20260928 08:42:05.533087 18687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "917b4b5757b5497c971a7efca3f35f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917b4b5757b5497c971a7efca3f35f86" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41859 } } }
I20260928 08:42:05.533453 18688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:05.533507 18687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:05.537144 18694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:05.542203 18694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:05.551359 18694 catalog_manager.cc:1347] Generated new cluster ID: 4c3bdfc3aab64f94aaac8423f6e069be
I20260928 08:42:05.551505 18694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:05.566679 18694 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:05.568025 18694 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:05.574357 18694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86: Generated new TSK 0
I20260928 08:42:05.575204 18694 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:42:05.642442 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41859
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:05.979022 18707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:05.979296 18707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:05.979390 18707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:05.986665 18707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:05.986771 18707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:05.986881 18707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:42:05.996806 18707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:05.998103 18707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:05.999437 18707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:06.012943 18713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:06.013015 18712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:06.013041 18715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:06.014782 18707 server_base.cc:1034] running on GCE node
I20260928 08:42:06.015497 18707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:06.016621 18707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:06.017874 18707 hybrid_clock.cc:648] HybridClock initialized: now 1790584926017814 us; error 52 us; skew 500 ppm
I20260928 08:42:06.018242 18707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:06 dist-test-slave-w6c0 krb5kdc[18601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:06.021616 18707 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:42:06.023358 18707 webserver.cc:460] Webserver started at http://127.0.97.1:33533/ using document root <none> and password file <none>
I20260928 08:42:06.024150 18707 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:06.024312 18707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:06.024637 18707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:06.027964 18707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "75c32a0d82a747a792a6cd223658b9bd"
format_stamp: "Formatted at 2026-09-28 08:42:06 on dist-test-slave-w6c0"
server_key: "7653f94b863b07be36b01640000eb78a"
server_key_iv: "773038f05b750c1eb05df263a7e04d09"
server_key_version: "encryptionkey@0"
I20260928 08:42:06.028781 18707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "75c32a0d82a747a792a6cd223658b9bd"
format_stamp: "Formatted at 2026-09-28 08:42:06 on dist-test-slave-w6c0"
server_key: "7653f94b863b07be36b01640000eb78a"
server_key_iv: "773038f05b750c1eb05df263a7e04d09"
server_key_version: "encryptionkey@0"
I20260928 08:42:06.034108 18707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:42:06.037737 18722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:06.039186 18707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:42:06.039383 18707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "75c32a0d82a747a792a6cd223658b9bd"
format_stamp: "Formatted at 2026-09-28 08:42:06 on dist-test-slave-w6c0"
server_key: "7653f94b863b07be36b01640000eb78a"
server_key_iv: "773038f05b750c1eb05df263a7e04d09"
server_key_version: "encryptionkey@0"
I20260928 08:42:06.039595 18707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:06.059093 18707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:06.063344 18707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:06.063673 18707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:06.064678 18707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:42:06.066062 18707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:42:06.066172 18707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:06.066319 18707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:42:06.066399 18707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:06.100685 18707 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40539
I20260928 08:42:06.100776 18834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40539 every 8 connection(s)
I20260928 08:42:06.101985 18707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:42:06.108151 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 18707
I20260928 08:42:06.108371 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:42:06.108817 388 external_mini_cluster.cc:1428] Setting key 5c79d361ac112d941c9a3c6a2a249da0
Sep 28 08:42:06 dist-test-slave-w6c0 krb5kdc[18601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:42:06.123608 18837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:06.104583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40891 (local address 127.0.97.62:41859)
0928 08:42:06.105061 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:06.105068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:06.105981 (+ 913us) server_negotiation.cc:407] Connection header received
0928 08:42:06.107846 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:06.107852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:06.107959 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:06.108093 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:06.109884 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:06.110752 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:06.112275 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:06.113011 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:06.113164 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:06.116845 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:06.116854 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:06.116861 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:06.116908 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:06.120126 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:06.121163 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:06.121168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:06.121170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:06.121302 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:06.121804 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:06.121807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:06.121809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:06.122078 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:06.122190 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:06.123166 (+ 976us) server_negotiation.cc:299] Negotiation successful
0928 08:42:06.123350 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":214,"threads_started":1}
I20260928 08:42:06.125116 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:06.105084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41859 (local address 127.0.97.1:40891)
0928 08:42:06.105808 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:06.105863 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:06.107574 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:06.108326 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:06.108342 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:06.108992 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:06.109698 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:06.109719 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:06.110921 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:06.110925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:06.112115 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:06.112127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:06.113213 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:06.113218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:06.113418 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:06.114132 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:42:06.114155 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:06.116597 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:06.120292 (+ 3695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:06.120302 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:06.120323 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:06.121018 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:06.121420 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:06.121428 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:06.121434 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:06.121693 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:06.122214 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:06.122225 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:06.122709 (+ 484us) client_negotiation.cc:769] Sending connection context
0928 08:42:06.123052 (+ 343us) client_negotiation.cc:240] Negotiation successful
0928 08:42:06.123656 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":276,"threads_started":1}
I20260928 08:42:06.126811 18835 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41859
I20260928 08:42:06.127138 18835 heartbeater.cc:464] Registering TS with master...
I20260928 08:42:06.127893 18835 heartbeater.cc:511] Master 127.0.97.62:41859 requested a full tablet report, sending...
I20260928 08:42:06.130060 18646 ts_manager.cc:194] Registered new tserver with Master: 75c32a0d82a747a792a6cd223658b9bd (127.0.97.1:40539)
I20260928 08:42:06.131811 18646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40891
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:42:06.170461 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41859
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:06.503501 18841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:06.503840 18841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:06.504011 18841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:06.511458 18841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:06.511620 18841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:06.511807 18841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:42:06.521628 18841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:06.522990 18841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:06.524359 18841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:06.537930 18849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:06.538007 18846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:06.537966 18847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:06.539264 18841 server_base.cc:1034] running on GCE node
I20260928 08:42:06.539849 18841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:06.540777 18841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:06.541985 18841 hybrid_clock.cc:648] HybridClock initialized: now 1790584926541944 us; error 27 us; skew 500 ppm
I20260928 08:42:06.542312 18841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:06 dist-test-slave-w6c0 krb5kdc[18601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:06.545637 18841 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:42:06.547067 18841 webserver.cc:460] Webserver started at http://127.0.97.2:44885/ using document root <none> and password file <none>
I20260928 08:42:06.547710 18841 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:06.547807 18841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:06.548066 18841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:06.550961 18841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "7cf945bd20984722a126c9816da58112"
format_stamp: "Formatted at 2026-09-28 08:42:06 on dist-test-slave-w6c0"
server_key: "a7ed2e66b77812ab0c0a99086e0a4916"
server_key_iv: "60c4251a18fdc91a156622d02449bf20"
server_key_version: "encryptionkey@0"
I20260928 08:42:06.551698 18841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7cf945bd20984722a126c9816da58112"
format_stamp: "Formatted at 2026-09-28 08:42:06 on dist-test-slave-w6c0"
server_key: "a7ed2e66b77812ab0c0a99086e0a4916"
server_key_iv: "60c4251a18fdc91a156622d02449bf20"
server_key_version: "encryptionkey@0"
I20260928 08:42:06.556802 18841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 08:42:06.560247 18856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:06.561553 18841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:42:06.561733 18841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "7cf945bd20984722a126c9816da58112"
format_stamp: "Formatted at 2026-09-28 08:42:06 on dist-test-slave-w6c0"
server_key: "a7ed2e66b77812ab0c0a99086e0a4916"
server_key_iv: "60c4251a18fdc91a156622d02449bf20"
server_key_version: "encryptionkey@0"
I20260928 08:42:06.561941 18841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:06.576056 18841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:06.580348 18841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:06.580672 18841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:06.581696 18841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:42:06.583140 18841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:42:06.583242 18841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:06.583396 18841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:42:06.583469 18841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:06.616276 18841 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45473
I20260928 08:42:06.616362 18968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45473 every 8 connection(s)
I20260928 08:42:06.617573 18841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:42:06.626435 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 18841
I20260928 08:42:06.626648 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:42:06.627218 388 external_mini_cluster.cc:1428] Setting key 8dc7044c9d5238812620b3224420633c
Sep 28 08:42:06 dist-test-slave-w6c0 krb5kdc[18601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:42:06.639681 18837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:06.620156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60675 (local address 127.0.97.62:41859)
0928 08:42:06.620350 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:06.620355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:06.621477 (+ 1122us) server_negotiation.cc:407] Connection header received
0928 08:42:06.623300 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:06.623306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:06.623385 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:06.623474 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:06.624956 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:06.625662 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:06.627873 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:06.628513 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:06.628619 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:06.632533 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:06.632541 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:06.632548 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:06.632602 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:06.636156 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:06.636979 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:06.637006 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:06.637012 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:06.637185 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:06.637593 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:06.637601 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:06.637607 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:06.637992 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:06.638141 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:06.638953 (+ 812us) server_negotiation.cc:299] Negotiation successful
0928 08:42:06.639232 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":76}
I20260928 08:42:06.640313 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:06.620607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41859 (local address 127.0.97.2:60675)
0928 08:42:06.621308 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:06.621351 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:06.623053 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:06.623690 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:06.623699 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:06.624269 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:06.624808 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:06.624824 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:06.625791 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:06.625794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:06.627673 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:06.627687 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:06.628740 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:06.628747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:06.629003 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:06.629828 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:42:06.629855 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:06.632313 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:06.636286 (+ 3973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:06.636293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:06.636304 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:06.636831 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:06.637294 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:06.637297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:06.637300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:06.637465 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:06.638152 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:06.638160 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:06.638492 (+ 332us) client_negotiation.cc:769] Sending connection context
0928 08:42:06.638784 (+ 292us) client_negotiation.cc:240] Negotiation successful
0928 08:42:06.639156 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":269,"threads_started":1}
I20260928 08:42:06.641716 18969 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41859
I20260928 08:42:06.642016 18969 heartbeater.cc:464] Registering TS with master...
I20260928 08:42:06.642808 18969 heartbeater.cc:511] Master 127.0.97.62:41859 requested a full tablet report, sending...
I20260928 08:42:06.644402 18646 ts_manager.cc:194] Registered new tserver with Master: 7cf945bd20984722a126c9816da58112 (127.0.97.2:45473)
I20260928 08:42:06.645218 18646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60675
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:42:06.683830 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41859
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:07.005136 18974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:07.005487 18974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:07.005674 18974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:07.013083 18974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:07.013248 18974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:07.013470 18974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:42:07.023530 18974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:07.024914 18974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:07.026242 18974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:07.040141 18982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:07.040180 18980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:07.040210 18979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:07.041002 18974 server_base.cc:1034] running on GCE node
I20260928 08:42:07.041611 18974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:07.042711 18974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:07.043970 18974 hybrid_clock.cc:648] HybridClock initialized: now 1790584927043874 us; error 80 us; skew 500 ppm
I20260928 08:42:07.044305 18974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:07 dist-test-slave-w6c0 krb5kdc[18601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:07.047592 18974 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:42:07.049145 18974 webserver.cc:460] Webserver started at http://127.0.97.3:36859/ using document root <none> and password file <none>
I20260928 08:42:07.049835 18974 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:07.049991 18974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:07.050323 18974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:07.053404 18974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "5971be383c1b4f5c929b3accca4f6a60"
format_stamp: "Formatted at 2026-09-28 08:42:07 on dist-test-slave-w6c0"
server_key: "4ce312d82dcb68fc2b5ad8fc6d9874e5"
server_key_iv: "9d6002b543e2055cb1c8d37116429a75"
server_key_version: "encryptionkey@0"
I20260928 08:42:07.054172 18974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5971be383c1b4f5c929b3accca4f6a60"
format_stamp: "Formatted at 2026-09-28 08:42:07 on dist-test-slave-w6c0"
server_key: "4ce312d82dcb68fc2b5ad8fc6d9874e5"
server_key_iv: "9d6002b543e2055cb1c8d37116429a75"
server_key_version: "encryptionkey@0"
I20260928 08:42:07.059365 18974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:42:07.062954 18989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:07.064282 18974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:42:07.064468 18974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "5971be383c1b4f5c929b3accca4f6a60"
format_stamp: "Formatted at 2026-09-28 08:42:07 on dist-test-slave-w6c0"
server_key: "4ce312d82dcb68fc2b5ad8fc6d9874e5"
server_key_iv: "9d6002b543e2055cb1c8d37116429a75"
server_key_version: "encryptionkey@0"
I20260928 08:42:07.064678 18974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:07.092821 18974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:07.096932 18974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:07.097183 18974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:07.098112 18974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:42:07.099483 18974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:42:07.099566 18974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:07.099648 18974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:42:07.099694 18974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:07.135031 18835 heartbeater.cc:503] Master 127.0.97.62:41859 was elected leader, sending a full tablet report...
I20260928 08:42:07.135565 18974 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45785
I20260928 08:42:07.135625 19101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45785 every 8 connection(s)
I20260928 08:42:07.136974 18974 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:42:07.146867 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 18974
I20260928 08:42:07.147037 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:42:07.147482 388 external_mini_cluster.cc:1428] Setting key 66c938f207e142d60170f2d647b25ecf
Sep 28 08:42:07 dist-test-slave-w6c0 krb5kdc[18601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:42:07.157966 18837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:07.139505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58311 (local address 127.0.97.62:41859)
0928 08:42:07.139718 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:07.139727 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:07.140966 (+ 1239us) server_negotiation.cc:407] Connection header received
0928 08:42:07.142938 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:07.142943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:07.143042 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:07.143163 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:07.144778 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.145562 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.147274 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.147772 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.147925 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:07.151870 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:07.151875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:07.151882 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:07.151919 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:07.154750 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:07.155622 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:07.155629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:07.155632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:07.155896 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:07.156370 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:07.156375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:07.156378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:07.156701 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:07.156833 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:07.157520 (+ 687us) server_negotiation.cc:299] Negotiation successful
0928 08:42:07.157694 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 08:42:07.158908 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:07.140043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41859 (local address 127.0.97.3:58311)
0928 08:42:07.140787 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:07.140828 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:07.142670 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:07.143451 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:07.143465 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:07.144060 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:07.144603 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:07.144620 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.145722 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.145728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:07.147127 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:07.147143 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.147994 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.148003 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:07.148249 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:07.149033 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:42:07.149056 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:07.151652 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:07.154894 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:07.154901 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:07.154913 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:07.155333 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:07.156034 (+ 701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:07.156037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:07.156039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:07.156208 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:07.156843 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:07.156848 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:07.157149 (+ 301us) client_negotiation.cc:769] Sending connection context
0928 08:42:07.157366 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 08:42:07.157750 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":314,"threads_started":1}
I20260928 08:42:07.160318 19102 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41859
I20260928 08:42:07.160630 19102 heartbeater.cc:464] Registering TS with master...
I20260928 08:42:07.161387 19102 heartbeater.cc:511] Master 127.0.97.62:41859 requested a full tablet report, sending...
I20260928 08:42:07.163450 18646 ts_manager.cc:194] Registered new tserver with Master: 5971be383c1b4f5c929b3accca4f6a60 (127.0.97.3:45785)
I20260928 08:42:07.164245 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:42:07.164783 18646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58311
I20260928 08:42:07.183779 18837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:07.170568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38682 (local address 127.0.97.62:41859)
0928 08:42:07.170727 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:07.170732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:07.171217 (+ 485us) server_negotiation.cc:407] Connection header received
0928 08:42:07.171334 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:07.171337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:07.171411 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:07.171497 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:07.172929 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.173624 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.175139 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.175531 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.175636 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:07.178219 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:07.178224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:07.178227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:07.178255 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:07.181302 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:07.182043 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:07.182047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:07.182049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:07.182165 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:07.182595 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:07.182599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:07.182601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:07.182877 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:07.183077 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:07.183397 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 08:42:07.183554 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 08:42:07.188387 18646 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:42:07.197045 18646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:42:07.231012 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:07.212335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43562 (local address 127.0.97.1:40539)
0928 08:42:07.222800 (+ 10465us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:07.222813 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:07.222851 (+ 38us) server_negotiation.cc:407] Connection header received
0928 08:42:07.222971 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:07.222980 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:07.223247 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:07.223813 (+ 566us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:07.224148 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.225743 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.228687 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.229777 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.229931 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:07.230227 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:07.230445 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 08:42:07.230640 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10067,"thread_start_us":9895,"threads_started":1}
I20260928 08:42:07.231115 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:07.212287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40539 (local address 127.0.0.1:43562)
0928 08:42:07.214456 (+ 2169us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:07.214529 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:07.214755 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:07.223483 (+ 8728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:07.223498 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:07.223530 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:07.224015 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:07.224032 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.225885 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.225891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:07.228500 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:07.228514 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.229972 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.229978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:07.230175 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:07.230232 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 08:42:07.230402 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 08:42:07.230640 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1848,"mutex_wait_us":21,"spinlock_wait_cycles":2816,"thread_start_us":10623,"threads_started":1}
I20260928 08:42:07.231868 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:07.214058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.0.97.2:45473)
0928 08:42:07.219059 (+ 5001us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:07.219072 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:07.219959 (+ 887us) server_negotiation.cc:407] Connection header received
0928 08:42:07.220098 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:07.220111 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:07.221392 (+ 1281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:07.221702 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:07.224632 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.226122 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.228460 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.229824 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.229951 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:07.230154 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:07.231382 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0928 08:42:07.231551 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4772,"thread_start_us":4645,"threads_started":1}
I20260928 08:42:07.232321 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:07.212569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45473 (local address 127.0.0.1:34120)
0928 08:42:07.219610 (+ 7041us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:07.219634 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:07.219802 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:07.223363 (+ 3561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:07.223370 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:07.223402 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:07.224427 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:07.224440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.226261 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.226267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:07.228305 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:07.228316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.230311 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.230319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:07.230475 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:07.230532 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 08:42:07.230616 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 08:42:07.230759 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6870,"thread_start_us":6675,"threads_started":1}
I20260928 08:42:07.232838 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:07.214834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56838 (local address 127.0.97.3:45785)
0928 08:42:07.215764 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:07.215777 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:07.223376 (+ 7599us) server_negotiation.cc:407] Connection header received
0928 08:42:07.223693 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:07.223701 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:07.223929 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:07.224105 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:07.225092 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.226699 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.229217 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.230626 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.230733 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:07.230932 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:07.232292 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0928 08:42:07.232477 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":505,"threads_started":1}
I20260928 08:42:07.233492 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:07.213575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45785 (local address 127.0.0.1:56838)
0928 08:42:07.223227 (+ 9652us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:07.223247 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:07.223476 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:07.224494 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:07.224500 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:07.224534 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:07.224901 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:07.224910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.226880 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.226886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:07.229041 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:07.229053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.231087 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.231097 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:07.232064 (+ 967us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:07.232108 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 08:42:07.233024 (+ 916us) client_negotiation.cc:240] Negotiation successful
0928 08:42:07.233157 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9501,"mutex_wait_us":25,"thread_start_us":1989,"threads_started":1}
I20260928 08:42:07.235616 18770 tablet_service.cc:1467] Processing CreateTablet for tablet ce113a4f7dd34c1abfdf541751ae9877 (DEFAULT_TABLE table=test-table [id=00337a9a04784ffd93a128245565cd7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:42:07.235616 18904 tablet_service.cc:1467] Processing CreateTablet for tablet ce113a4f7dd34c1abfdf541751ae9877 (DEFAULT_TABLE table=test-table [id=00337a9a04784ffd93a128245565cd7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:42:07.237078 18904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce113a4f7dd34c1abfdf541751ae9877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:07.237078 18770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce113a4f7dd34c1abfdf541751ae9877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:07.239107 19037 tablet_service.cc:1467] Processing CreateTablet for tablet ce113a4f7dd34c1abfdf541751ae9877 (DEFAULT_TABLE table=test-table [id=00337a9a04784ffd93a128245565cd7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:42:07.240231 19037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce113a4f7dd34c1abfdf541751ae9877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:07.251084 19120 tablet_bootstrap.cc:492] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60: Bootstrap starting.
I20260928 08:42:07.254468 19120 tablet_bootstrap.cc:654] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:07.255534 19120 log.cc:824] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:07.256901 19121 tablet_bootstrap.cc:492] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd: Bootstrap starting.
I20260928 08:42:07.258880 19120 tablet_bootstrap.cc:492] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60: No bootstrap required, opened a new log
I20260928 08:42:07.258960 19122 tablet_bootstrap.cc:492] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112: Bootstrap starting.
I20260928 08:42:07.259243 19120 ts_tablet_manager.cc:1397] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 08:42:07.262447 19122 tablet_bootstrap.cc:654] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:07.263695 19122 log.cc:824] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:07.264902 19121 tablet_bootstrap.cc:654] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:07.265980 19121 log.cc:824] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:07.266849 19122 tablet_bootstrap.cc:492] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112: No bootstrap required, opened a new log
I20260928 08:42:07.267109 19122 ts_tablet_manager.cc:1397] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260928 08:42:07.268813 19121 tablet_bootstrap.cc:492] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd: No bootstrap required, opened a new log
I20260928 08:42:07.269124 19121 ts_tablet_manager.cc:1397] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260928 08:42:07.270032 19120 raft_consensus.cc:348] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c32a0d82a747a792a6cd223658b9bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40539 } } peers { permanent_uuid: "7cf945bd20984722a126c9816da58112" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } } peers { permanent_uuid: "5971be383c1b4f5c929b3accca4f6a60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45785 } }
I20260928 08:42:07.270447 19120 raft_consensus.cc:374] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:07.270552 19120 raft_consensus.cc:729] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5971be383c1b4f5c929b3accca4f6a60, State: Initialized, Role: FOLLOWER
I20260928 08:42:07.271193 19120 consensus_queue.cc:260] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c32a0d82a747a792a6cd223658b9bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40539 } } peers { permanent_uuid: "7cf945bd20984722a126c9816da58112" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } } peers { permanent_uuid: "5971be383c1b4f5c929b3accca4f6a60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45785 } }
I20260928 08:42:07.272259 19102 heartbeater.cc:503] Master 127.0.97.62:41859 was elected leader, sending a full tablet report...
I20260928 08:42:07.273216 19120 ts_tablet_manager.cc:1428] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
I20260928 08:42:07.277741 19122 raft_consensus.cc:348] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c32a0d82a747a792a6cd223658b9bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40539 } } peers { permanent_uuid: "7cf945bd20984722a126c9816da58112" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } } peers { permanent_uuid: "5971be383c1b4f5c929b3accca4f6a60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45785 } }
I20260928 08:42:07.278209 19122 raft_consensus.cc:374] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:07.278326 19122 raft_consensus.cc:729] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cf945bd20984722a126c9816da58112, State: Initialized, Role: FOLLOWER
I20260928 08:42:07.278936 19122 consensus_queue.cc:260] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c32a0d82a747a792a6cd223658b9bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40539 } } peers { permanent_uuid: "7cf945bd20984722a126c9816da58112" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } } peers { permanent_uuid: "5971be383c1b4f5c929b3accca4f6a60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45785 } }
I20260928 08:42:07.279028 19121 raft_consensus.cc:348] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c32a0d82a747a792a6cd223658b9bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40539 } } peers { permanent_uuid: "7cf945bd20984722a126c9816da58112" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } } peers { permanent_uuid: "5971be383c1b4f5c929b3accca4f6a60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45785 } }
I20260928 08:42:07.279382 19121 raft_consensus.cc:374] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:07.279496 19121 raft_consensus.cc:729] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c32a0d82a747a792a6cd223658b9bd, State: Initialized, Role: FOLLOWER
I20260928 08:42:07.280117 19121 consensus_queue.cc:260] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c32a0d82a747a792a6cd223658b9bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40539 } } peers { permanent_uuid: "7cf945bd20984722a126c9816da58112" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } } peers { permanent_uuid: "5971be383c1b4f5c929b3accca4f6a60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45785 } }
I20260928 08:42:07.280627 18969 heartbeater.cc:503] Master 127.0.97.62:41859 was elected leader, sending a full tablet report...
I20260928 08:42:07.281198 19122 ts_tablet_manager.cc:1428] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260928 08:42:07.281749 19121 ts_tablet_manager.cc:1428] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
W20260928 08:42:07.354486 18836 tablet.cc:2369] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:42:07.369585 18970 tablet.cc:2369] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:42:07.388763 19103 tablet.cc:2369] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:42:07.400107 19128 raft_consensus.cc:482] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:42:07.400731 19128 raft_consensus.cc:504] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c32a0d82a747a792a6cd223658b9bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40539 } } peers { permanent_uuid: "7cf945bd20984722a126c9816da58112" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } } peers { permanent_uuid: "5971be383c1b4f5c929b3accca4f6a60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45785 } }
I20260928 08:42:07.402766 19128 leader_election.cc:288] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cf945bd20984722a126c9816da58112 (127.0.97.2:45473), 5971be383c1b4f5c929b3accca4f6a60 (127.0.97.3:45785)
I20260928 08:42:07.409992 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:07.403283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46465 (local address 127.0.97.3:45785)
0928 08:42:07.403503 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:07.403510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:07.404300 (+ 790us) server_negotiation.cc:407] Connection header received
0928 08:42:07.404500 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:07.404504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:07.404578 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:07.404686 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:07.405367 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.406108 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.408106 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.408870 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.409032 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:07.409139 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:07.409608 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 08:42:07.409739 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 08:42:07.410085 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:07.403233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48131 (local address 127.0.97.2:45473)
0928 08:42:07.403489 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:07.403499 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:07.404401 (+ 902us) server_negotiation.cc:407] Connection header received
0928 08:42:07.404597 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:07.404603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:07.404674 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:07.404773 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:07.405371 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.406104 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.408087 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.408852 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.409021 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:07.409113 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:07.409707 (+ 594us) server_negotiation.cc:299] Negotiation successful
0928 08:42:07.409838 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:42:07.410228 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:07.403270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45473 (local address 127.0.97.1:48131)
0928 08:42:07.404289 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:07.404307 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:07.404446 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:07.404786 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:07.404794 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:07.404843 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:07.405204 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:07.405214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.406239 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.406244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:07.407929 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:07.407941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.409298 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.409306 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:07.409552 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:07.409589 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 08:42:07.409681 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 08:42:07.409805 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":55,"spinlock_wait_cycles":28800,"thread_start_us":548,"threads_started":1}
I20260928 08:42:07.410816 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:07.403175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45785 (local address 127.0.97.1:46465)
0928 08:42:07.404180 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:07.404196 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:07.404331 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:07.404705 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:07.404710 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:07.404735 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:07.405207 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:07.405216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.406248 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.406255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:07.407938 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:07.407951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.409112 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.409119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:07.409294 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:07.409433 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 08:42:07.410255 (+ 822us) client_negotiation.cc:240] Negotiation successful
0928 08:42:07.410408 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":84,"spinlock_wait_cycles":2816,"thread_start_us":613,"threads_started":1}
I20260928 08:42:07.411321 18924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce113a4f7dd34c1abfdf541751ae9877" candidate_uuid: "75c32a0d82a747a792a6cd223658b9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cf945bd20984722a126c9816da58112" is_pre_election: true
I20260928 08:42:07.411713 19057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce113a4f7dd34c1abfdf541751ae9877" candidate_uuid: "75c32a0d82a747a792a6cd223658b9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5971be383c1b4f5c929b3accca4f6a60" is_pre_election: true
I20260928 08:42:07.411856 18924 raft_consensus.cc:2454] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75c32a0d82a747a792a6cd223658b9bd in term 0.
I20260928 08:42:07.412181 19057 raft_consensus.cc:2454] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75c32a0d82a747a792a6cd223658b9bd in term 0.
I20260928 08:42:07.412566 18724 leader_election.cc:302] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75c32a0d82a747a792a6cd223658b9bd, 7cf945bd20984722a126c9816da58112; no voters:
I20260928 08:42:07.412977 19128 raft_consensus.cc:2781] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:42:07.413127 19128 raft_consensus.cc:482] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:42:07.413244 19128 raft_consensus.cc:3037] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:07.415570 19128 raft_consensus.cc:504] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c32a0d82a747a792a6cd223658b9bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40539 } } peers { permanent_uuid: "7cf945bd20984722a126c9816da58112" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } } peers { permanent_uuid: "5971be383c1b4f5c929b3accca4f6a60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45785 } }
I20260928 08:42:07.416246 19128 leader_election.cc:288] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [CANDIDATE]: Term 1 election: Requested vote from peers 7cf945bd20984722a126c9816da58112 (127.0.97.2:45473), 5971be383c1b4f5c929b3accca4f6a60 (127.0.97.3:45785)
I20260928 08:42:07.416709 19057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce113a4f7dd34c1abfdf541751ae9877" candidate_uuid: "75c32a0d82a747a792a6cd223658b9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5971be383c1b4f5c929b3accca4f6a60"
I20260928 08:42:07.416707 18924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce113a4f7dd34c1abfdf541751ae9877" candidate_uuid: "75c32a0d82a747a792a6cd223658b9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cf945bd20984722a126c9816da58112"
I20260928 08:42:07.416913 18924 raft_consensus.cc:3037] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:07.416927 19057 raft_consensus.cc:3037] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:07.419183 19057 raft_consensus.cc:2454] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75c32a0d82a747a792a6cd223658b9bd in term 1.
I20260928 08:42:07.419186 18924 raft_consensus.cc:2454] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75c32a0d82a747a792a6cd223658b9bd in term 1.
I20260928 08:42:07.419581 18726 leader_election.cc:302] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5971be383c1b4f5c929b3accca4f6a60, 75c32a0d82a747a792a6cd223658b9bd; no voters:
I20260928 08:42:07.419925 19128 raft_consensus.cc:2781] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:07.420369 19128 raft_consensus.cc:686] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [term 1 LEADER]: Becoming Leader. State: Replica: 75c32a0d82a747a792a6cd223658b9bd, State: Running, Role: LEADER
I20260928 08:42:07.420930 19128 consensus_queue.cc:237] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c32a0d82a747a792a6cd223658b9bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40539 } } peers { permanent_uuid: "7cf945bd20984722a126c9816da58112" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } } peers { permanent_uuid: "5971be383c1b4f5c929b3accca4f6a60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45785 } }
I20260928 08:42:07.425416 18645 catalog_manager.cc:5495] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 75c32a0d82a747a792a6cd223658b9bd (127.0.97.1). New cstate: current_term: 1 leader_uuid: "75c32a0d82a747a792a6cd223658b9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c32a0d82a747a792a6cd223658b9bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7cf945bd20984722a126c9816da58112" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5971be383c1b4f5c929b3accca4f6a60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45785 } health_report { overall_health: UNKNOWN } } }
I20260928 08:42:07.468855 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:07.463310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43572 (local address 127.0.97.1:40539)
0928 08:42:07.463517 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:07.463522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:07.463540 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:42:07.463669 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:07.463672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:07.463761 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:07.463883 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:42:07.464238 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.464972 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.466543 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.466938 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.467041 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:07.467641 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:42:07.468197 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:42:07.468335 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:07.468510 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 08:42:07.468578 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:42:07.477794 18924 raft_consensus.cc:1261] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112 [term 1 FOLLOWER]: Refusing update from remote peer 75c32a0d82a747a792a6cd223658b9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:42:07.477794 19057 raft_consensus.cc:1261] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60 [term 1 FOLLOWER]: Refusing update from remote peer 75c32a0d82a747a792a6cd223658b9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:42:07.479089 19131 consensus_queue.cc:1035] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5971be383c1b4f5c929b3accca4f6a60" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:42:07.479471 19128 consensus_queue.cc:1035] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cf945bd20984722a126c9816da58112" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:42:07.497792 19134 mvcc.cc:205] Tried to move back new op lower bound from 7334235862938525696 to 7334235862720335872. Current Snapshot: MvccSnapshot[applied={T|T < 7334235862938525696}]
I20260928 08:42:07.499603 19135 mvcc.cc:205] Tried to move back new op lower bound from 7334235862938525696 to 7334235862720335872. Current Snapshot: MvccSnapshot[applied={T|T < 7334235862938525696}]
I20260928 08:42:07.516666 18645 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:42:07.517017 18645 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:42:07.522045 18645 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 917b4b5757b5497c971a7efca3f35f86: Sending DeleteTablet for 3 replicas of tablet ce113a4f7dd34c1abfdf541751ae9877
I20260928 08:42:07.523386 18904 tablet_service.cc:1514] Processing DeleteTablet for tablet ce113a4f7dd34c1abfdf541751ae9877 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:42:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34120
I20260928 08:42:07.523809 18770 tablet_service.cc:1514] Processing DeleteTablet for tablet ce113a4f7dd34c1abfdf541751ae9877 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:42:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43562
I20260928 08:42:07.524119 19037 tablet_service.cc:1514] Processing DeleteTablet for tablet ce113a4f7dd34c1abfdf541751ae9877 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:42:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56838
I20260928 08:42:07.529199 19145 tablet_replica.cc:330] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60: stopping tablet replica
I20260928 08:42:07.529641 19145 raft_consensus.cc:2229] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:42:07.530701 19144 tablet_replica.cc:330] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd: stopping tablet replica
I20260928 08:42:07.531111 19144 raft_consensus.cc:2229] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:42:07.531726 19143 tablet_replica.cc:330] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112: stopping tablet replica
I20260928 08:42:07.532094 19143 raft_consensus.cc:2229] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:42:07.532286 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 18707
I20260928 08:42:07.532572 19143 raft_consensus.cc:2258] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:42:07.532882 19145 raft_consensus.cc:2258] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:42:07.533715 19144 raft_consensus.cc:2258] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:42:07.537189 19145 ts_tablet_manager.cc:1905] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:42:07.537190 19143 ts_tablet_manager.cc:1905] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:42:07.539836 19144 ts_tablet_manager.cc:1905] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:42:07.544646 19143 ts_tablet_manager.cc:1918] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:42:07.544760 19145 ts_tablet_manager.cc:1918] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:42:07.544818 19143 log.cc:1196] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/ce113a4f7dd34c1abfdf541751ae9877
I20260928 08:42:07.544945 19145 log.cc:1196] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/ce113a4f7dd34c1abfdf541751ae9877
I20260928 08:42:07.545450 19143 ts_tablet_manager.cc:1939] T ce113a4f7dd34c1abfdf541751ae9877 P 7cf945bd20984722a126c9816da58112: Deleting consensus metadata
I20260928 08:42:07.545503 19145 ts_tablet_manager.cc:1939] T ce113a4f7dd34c1abfdf541751ae9877 P 5971be383c1b4f5c929b3accca4f6a60: Deleting consensus metadata
I20260928 08:42:07.547065 19144 ts_tablet_manager.cc:1918] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:42:07.547217 19144 log.cc:1196] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584618551208-388-0/minicluster-data/ts-0/wal/wals/ce113a4f7dd34c1abfdf541751ae9877
Sep 28 08:42:07 dist-test-slave-w6c0 krb5kdc[18601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:42:07.547417 18631 catalog_manager.cc:4841] TS 7cf945bd20984722a126c9816da58112 (127.0.97.2:45473): tablet ce113a4f7dd34c1abfdf541751ae9877 (table test-table [id=00337a9a04784ffd93a128245565cd7d]) successfully deleted
I20260928 08:42:07.547757 19144 ts_tablet_manager.cc:1939] T ce113a4f7dd34c1abfdf541751ae9877 P 75c32a0d82a747a792a6cd223658b9bd: Deleting consensus metadata
I20260928 08:42:07.547770 18633 catalog_manager.cc:4841] TS 5971be383c1b4f5c929b3accca4f6a60 (127.0.97.3:45785): tablet ce113a4f7dd34c1abfdf541751ae9877 (table test-table [id=00337a9a04784ffd93a128245565cd7d]) successfully deleted
I20260928 08:42:07.549409 18630 catalog_manager.cc:4841] TS 75c32a0d82a747a792a6cd223658b9bd (127.0.97.1:40539): tablet ce113a4f7dd34c1abfdf541751ae9877 (table test-table [id=00337a9a04784ffd93a128245565cd7d]) successfully deleted
I20260928 08:42:07.555132 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:07.533633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43584 (local address 127.0.97.1:40539)
0928 08:42:07.533858 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:07.533864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:07.535729 (+ 1865us) server_negotiation.cc:407] Connection header received
0928 08:42:07.536032 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:07.536038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:07.536130 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:07.536215 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:07.538336 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.539479 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.541494 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.542209 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.542311 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:07.548776 (+ 6465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:07.548788 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:07.548802 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:07.548848 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:07.552196 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:07.552953 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:07.552961 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:07.552964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:07.553110 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:07.553510 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:07.553517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:07.553524 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:07.553868 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:07.554202 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:07.554490 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 08:42:07.554789 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:42:07.555886 18830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:07.657083 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 18707
I20260928 08:42:07.675575 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 18841
Sep 28 08:42:07 dist-test-slave-w6c0 krb5kdc[18601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:42:07.691383 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:07.676363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34132 (local address 127.0.97.2:45473)
0928 08:42:07.676539 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:07.676547 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:07.676568 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:42:07.676792 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:07.676796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:07.676879 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:07.676988 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:07.678216 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.679030 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.680311 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.680697 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.680839 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:07.685494 (+ 4655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:07.685509 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:07.685525 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:07.685579 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:07.688469 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:07.689152 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:07.689159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:07.689161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:07.689294 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:07.689677 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:07.689686 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:07.689688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:07.690096 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:07.690401 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:07.690833 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 08:42:07.691068 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:42:07.692178 18964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:07.783857 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 18841
I20260928 08:42:07.801816 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 18974
Sep 28 08:42:07 dist-test-slave-w6c0 krb5kdc[18601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:42:07.817790 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:07.803100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56854 (local address 127.0.97.3:45785)
0928 08:42:07.803347 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:07.803353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:07.803372 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:42:07.803429 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:07.803434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:07.803500 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:07.803586 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:07.805001 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.805822 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.807179 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:07.807574 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:07.807674 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:07.812012 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:07.812026 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:07.812043 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:07.812082 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:07.815003 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:07.815603 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:07.815610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:07.815613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:07.815752 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:07.816481 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:07.816485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:07.816487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:07.816824 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:07.817046 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:07.817375 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 08:42:07.817534 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 08:42:07.818492 19097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:07.909940 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 18974
I20260928 08:42:07.927414 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 18614
I20260928 08:42:07.928566 18676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:08.042418 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 18614
2026-09-28T08:42:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4340 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T08:42:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:08Z Disabled control of system clock
I20260928 08:42:08.120654 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36933
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:42:08.455286 19154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:08.455582 19154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:08.455663 19154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:08.463092 19154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:08.463207 19154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:08.463245 19154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:08.463299 19154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:08.463343 19154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:08.472610 19154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36933
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:08.473930 19154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:08.475755 19154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:08.487429 19159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:08.487360 19162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:08.487361 19160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:08.488089 19154 server_base.cc:1034] running on GCE node
I20260928 08:42:08.488865 19154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:08.490266 19154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:08.491465 19154 hybrid_clock.cc:648] HybridClock initialized: now 1790584928491425 us; error 42 us; skew 500 ppm
I20260928 08:42:08.491832 19154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:42:08.494181 19154 webserver.cc:460] Webserver started at http://127.0.97.62:33235/ using document root <none> and password file <none>
I20260928 08:42:08.494910 19154 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:08.494995 19154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:08.495270 19154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:08.498052 19154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "232d6c7405ca41f6a34ed7d921219d4c"
format_stamp: "Formatted at 2026-09-28 08:42:08 on dist-test-slave-w6c0"
server_key: "b9e4da437076aec4b776a6e6a71f1c17"
server_key_iv: "e8c077a694525ce016bbf772bc8f7891"
server_key_version: "encryptionkey@0"
I20260928 08:42:08.498862 19154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "232d6c7405ca41f6a34ed7d921219d4c"
format_stamp: "Formatted at 2026-09-28 08:42:08 on dist-test-slave-w6c0"
server_key: "b9e4da437076aec4b776a6e6a71f1c17"
server_key_iv: "e8c077a694525ce016bbf772bc8f7891"
server_key_version: "encryptionkey@0"
I20260928 08:42:08.503764 19154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 08:42:08.506989 19168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:08.508366 19154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 08:42:08.508543 19154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "232d6c7405ca41f6a34ed7d921219d4c"
format_stamp: "Formatted at 2026-09-28 08:42:08 on dist-test-slave-w6c0"
server_key: "b9e4da437076aec4b776a6e6a71f1c17"
server_key_iv: "e8c077a694525ce016bbf772bc8f7891"
server_key_version: "encryptionkey@0"
I20260928 08:42:08.508767 19154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:08.536219 19154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:08.537096 19154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:08.537395 19154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:08.555745 19154 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36933
I20260928 08:42:08.555752 19219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36933 every 8 connection(s)
I20260928 08:42:08.557094 19154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:08.560760 19220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:08.564507 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 19154
I20260928 08:42:08.564704 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:08.565219 388 external_mini_cluster.cc:1428] Setting key 93cef0695a5c84ee9d5c8ccc8d35363d
I20260928 08:42:08.573143 19220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c: Bootstrap starting.
I20260928 08:42:08.576447 19220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:08.577647 19220 log.cc:824] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:08.578610 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:08.567126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.0.97.62:36933)
0928 08:42:08.568083 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:08.568093 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:08.568134 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:42:08.569756 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:08.569776 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:08.570232 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:08.570646 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:08.571205 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:08.572437 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:08.573851 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:08.574592 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:08.574798 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:08.575971 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:08.575978 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:08.575991 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:08.576566 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:08.576674 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:08.576697 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:08.576833 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:08.577048 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 08:42:08.577372 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":355,"threads_started":1}
I20260928 08:42:08.580883 19220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c: No bootstrap required, opened a new log
I20260928 08:42:08.588634 19220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "232d6c7405ca41f6a34ed7d921219d4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36933 } }
I20260928 08:42:08.589043 19220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:08.589159 19220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 232d6c7405ca41f6a34ed7d921219d4c, State: Initialized, Role: FOLLOWER
I20260928 08:42:08.589809 19220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "232d6c7405ca41f6a34ed7d921219d4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36933 } }
I20260928 08:42:08.590116 19220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:08.590265 19220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:08.590441 19220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:08.592655 19220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "232d6c7405ca41f6a34ed7d921219d4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36933 } }
I20260928 08:42:08.593245 19220 leader_election.cc:302] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 232d6c7405ca41f6a34ed7d921219d4c; no voters:
I20260928 08:42:08.593919 19220 leader_election.cc:288] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:08.594058 19225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:08.594962 19225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c [term 1 LEADER]: Becoming Leader. State: Replica: 232d6c7405ca41f6a34ed7d921219d4c, State: Running, Role: LEADER
I20260928 08:42:08.595532 19225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "232d6c7405ca41f6a34ed7d921219d4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36933 } }
I20260928 08:42:08.596360 19220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:08.598739 19226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "232d6c7405ca41f6a34ed7d921219d4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "232d6c7405ca41f6a34ed7d921219d4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36933 } } }
I20260928 08:42:08.599611 19226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:08.598765 19227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 232d6c7405ca41f6a34ed7d921219d4c. Latest consensus state: current_term: 1 leader_uuid: "232d6c7405ca41f6a34ed7d921219d4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "232d6c7405ca41f6a34ed7d921219d4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36933 } } }
I20260928 08:42:08.600092 19227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:08.603569 19232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:08.607820 19232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:08.616333 19232 catalog_manager.cc:1347] Generated new cluster ID: be1055d2415e4494be836661dfd7bcb7
I20260928 08:42:08.616463 19232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:08.625962 19232 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:08.627452 19232 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:08.633528 19232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c: Generated new TSK 0
I20260928 08:42:08.634294 19232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:08.641335 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36933
--builtin_ntp_servers=127.0.97.20:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:42:08.991488 19244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:08.991863 19244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:08.992043 19244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:09.000021 19244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:09.000173 19244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:09.000356 19244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:42:09.010180 19244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:09.011693 19244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:09.013758 19244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:09.028535 19250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:09.028434 19252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:09.028471 19249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:09.029590 19244 server_base.cc:1034] running on GCE node
I20260928 08:42:09.030366 19244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:09.031447 19244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:09.032718 19244 hybrid_clock.cc:648] HybridClock initialized: now 1790584929032653 us; error 42 us; skew 500 ppm
I20260928 08:42:09.033084 19244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:42:09.035554 19244 webserver.cc:460] Webserver started at http://127.0.97.1:35899/ using document root <none> and password file <none>
I20260928 08:42:09.036305 19244 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:09.036448 19244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:09.036782 19244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:09.039898 19244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "14f7ab021ed84316b12fed5f6f04f6b2"
format_stamp: "Formatted at 2026-09-28 08:42:09 on dist-test-slave-w6c0"
server_key: "69a553c5692c2e251e05568c712f73ee"
server_key_iv: "081dccd7ed5ca64d8f675e2a182f8327"
server_key_version: "encryptionkey@0"
I20260928 08:42:09.040716 19244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "14f7ab021ed84316b12fed5f6f04f6b2"
format_stamp: "Formatted at 2026-09-28 08:42:09 on dist-test-slave-w6c0"
server_key: "69a553c5692c2e251e05568c712f73ee"
server_key_iv: "081dccd7ed5ca64d8f675e2a182f8327"
server_key_version: "encryptionkey@0"
I20260928 08:42:09.046670 19244 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 08:42:09.050338 19258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:09.052350 19244 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 08:42:09.052682 19244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "14f7ab021ed84316b12fed5f6f04f6b2"
format_stamp: "Formatted at 2026-09-28 08:42:09 on dist-test-slave-w6c0"
server_key: "69a553c5692c2e251e05568c712f73ee"
server_key_iv: "081dccd7ed5ca64d8f675e2a182f8327"
server_key_version: "encryptionkey@0"
I20260928 08:42:09.052994 19244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:09.073076 19244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:09.074090 19244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:09.074411 19244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:09.075610 19244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:42:09.077226 19244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:42:09.077385 19244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:09.077528 19244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:42:09.077621 19244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:09.114111 19244 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40383
I20260928 08:42:09.114174 19370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40383 every 8 connection(s)
I20260928 08:42:09.115506 19244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:42:09.120575 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 19244
I20260928 08:42:09.120805 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:42:09.121436 388 external_mini_cluster.cc:1428] Setting key 438f79ef4306040f342f7ca65b0559c4
I20260928 08:42:09.124950 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36933
--builtin_ntp_servers=127.0.97.20:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 08:42:09.134027 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:09.119075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57177 (local address 127.0.97.62:36933)
0928 08:42:09.119739 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:09.119747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:09.121458 (+ 1711us) server_negotiation.cc:407] Connection header received
0928 08:42:09.125370 (+ 3912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:09.125378 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:09.125490 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:09.125618 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:09.127096 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:09.128259 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:09.129943 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:09.130738 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:09.130881 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:09.132688 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:09.132697 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:09.132710 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:09.132754 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:09.132814 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:09.132833 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:09.132961 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:09.133488 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 08:42:09.133625 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":271,"threads_started":1}
I20260928 08:42:09.135517 19374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:09.119775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36933 (local address 127.0.97.1:57177)
0928 08:42:09.121246 (+ 1471us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:09.121294 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:09.123701 (+ 2407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:09.125925 (+ 2224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:09.125935 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:09.126058 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:09.126886 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:09.126907 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:09.128444 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:09.128448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:09.129770 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:09.129783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:09.131022 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:09.131028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:09.131261 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:09.132266 (+ 1005us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:42:09.132268 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:09.132312 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:42:09.132401 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:09.132983 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:09.132996 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:09.133008 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 08:42:09.133464 (+ 456us) client_negotiation.cc:240] Negotiation successful
0928 08:42:09.133932 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":475,"threads_started":1}
I20260928 08:42:09.137775 19371 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36933
I20260928 08:42:09.138162 19371 heartbeater.cc:464] Registering TS with master...
I20260928 08:42:09.139084 19371 heartbeater.cc:511] Master 127.0.97.62:36933 requested a full tablet report, sending...
I20260928 08:42:09.141976 19185 ts_manager.cc:194] Registered new tserver with Master: 14f7ab021ed84316b12fed5f6f04f6b2 (127.0.97.1:40383)
I20260928 08:42:09.143942 19185 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:57177
W20260928 08:42:09.475602 19375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:09.475996 19375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:09.476159 19375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:09.483853 19375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:09.484026 19375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:09.484212 19375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:42:09.493922 19375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:09.495344 19375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:09.497383 19375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:09.512130 19383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:09.512173 19381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:09.512220 19380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:09.516011 19375 server_base.cc:1034] running on GCE node
I20260928 08:42:09.516706 19375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:09.517684 19375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:09.518887 19375 hybrid_clock.cc:648] HybridClock initialized: now 1790584929518823 us; error 62 us; skew 500 ppm
I20260928 08:42:09.519219 19375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:42:09.521576 19375 webserver.cc:460] Webserver started at http://127.0.97.2:46439/ using document root <none> and password file <none>
I20260928 08:42:09.522267 19375 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:09.522351 19375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:09.522732 19375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:09.525669 19375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "a54227eba6144f448cf857b04000c182"
format_stamp: "Formatted at 2026-09-28 08:42:09 on dist-test-slave-w6c0"
server_key: "aeb987245d35dfcdf16cac9122946960"
server_key_iv: "e1defa8e15058a8ed2fff017edb95d29"
server_key_version: "encryptionkey@0"
I20260928 08:42:09.526422 19375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a54227eba6144f448cf857b04000c182"
format_stamp: "Formatted at 2026-09-28 08:42:09 on dist-test-slave-w6c0"
server_key: "aeb987245d35dfcdf16cac9122946960"
server_key_iv: "e1defa8e15058a8ed2fff017edb95d29"
server_key_version: "encryptionkey@0"
I20260928 08:42:09.531783 19375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:42:09.535585 19389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:09.536940 19375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:42:09.537073 19375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "a54227eba6144f448cf857b04000c182"
format_stamp: "Formatted at 2026-09-28 08:42:09 on dist-test-slave-w6c0"
server_key: "aeb987245d35dfcdf16cac9122946960"
server_key_iv: "e1defa8e15058a8ed2fff017edb95d29"
server_key_version: "encryptionkey@0"
I20260928 08:42:09.537267 19375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:09.573771 19375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:09.574666 19375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:09.574910 19375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:09.575853 19375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:42:09.577312 19375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:42:09.577399 19375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:09.577478 19375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:42:09.577534 19375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:09.611344 19375 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34949
I20260928 08:42:09.611446 19501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34949 every 8 connection(s)
I20260928 08:42:09.612884 19375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:42:09.614758 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 19375
I20260928 08:42:09.614914 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:42:09.615530 388 external_mini_cluster.cc:1428] Setting key 8493ad0e771ff5e7db4686bb08be434a
I20260928 08:42:09.618490 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36933
--builtin_ntp_servers=127.0.97.20:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 08:42:09.627051 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:09.615797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41197 (local address 127.0.97.62:36933)
0928 08:42:09.616007 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:09.616012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:09.617717 (+ 1705us) server_negotiation.cc:407] Connection header received
0928 08:42:09.619694 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:09.619703 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:09.619796 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:09.619912 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:09.620938 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:09.622082 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:09.623984 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:09.624684 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:09.624828 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:09.626200 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:09.626209 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:09.626217 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:09.626248 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:09.626294 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:09.626308 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:09.626397 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:09.626702 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 08:42:09.626793 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 08:42:09.628306 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:09.616457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36933 (local address 127.0.97.2:41197)
0928 08:42:09.617561 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:09.617604 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:09.619412 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:09.620079 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:09.620087 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:09.620173 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:09.620761 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:09.620777 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:09.622256 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:09.622260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:09.623815 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:09.623829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:09.624890 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:09.624895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:09.625098 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:09.625861 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:42:09.625864 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:09.625907 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:42:09.625995 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:09.626433 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:09.626444 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:09.626454 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 08:42:09.626690 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 08:42:09.627153 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":374,"threads_started":1}
I20260928 08:42:09.629882 19502 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36933
I20260928 08:42:09.630224 19502 heartbeater.cc:464] Registering TS with master...
I20260928 08:42:09.631163 19502 heartbeater.cc:511] Master 127.0.97.62:36933 requested a full tablet report, sending...
I20260928 08:42:09.632815 19185 ts_manager.cc:194] Registered new tserver with Master: a54227eba6144f448cf857b04000c182 (127.0.97.2:34949)
I20260928 08:42:09.633817 19185 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:41197
W20260928 08:42:09.960711 19505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:09.961061 19505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:09.961221 19505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:09.968863 19505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:09.969017 19505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:09.969202 19505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:42:09.978871 19505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:09.980289 19505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:09.982149 19505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:09.995666 19510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:09.995627 19511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:09.995595 19513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:09.996551 19505 server_base.cc:1034] running on GCE node
I20260928 08:42:09.997269 19505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:09.998458 19505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:09.999729 19505 hybrid_clock.cc:648] HybridClock initialized: now 1790584929999655 us; error 68 us; skew 500 ppm
I20260928 08:42:10.000079 19505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:42:10.002621 19505 webserver.cc:460] Webserver started at http://127.0.97.3:42829/ using document root <none> and password file <none>
I20260928 08:42:10.003350 19505 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:10.003463 19505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:10.003808 19505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:10.006824 19505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "82cca18ab819496ca9c61446a2baba9b"
format_stamp: "Formatted at 2026-09-28 08:42:10 on dist-test-slave-w6c0"
server_key: "8219c018ca4afda26461ce5eccea4557"
server_key_iv: "aee8a0d8a183f7267475e0c8cc8b25da"
server_key_version: "encryptionkey@0"
I20260928 08:42:10.007639 19505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "82cca18ab819496ca9c61446a2baba9b"
format_stamp: "Formatted at 2026-09-28 08:42:10 on dist-test-slave-w6c0"
server_key: "8219c018ca4afda26461ce5eccea4557"
server_key_iv: "aee8a0d8a183f7267475e0c8cc8b25da"
server_key_version: "encryptionkey@0"
I20260928 08:42:10.013098 19505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:42:10.016860 19519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:10.018386 19505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:42:10.018610 19505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "82cca18ab819496ca9c61446a2baba9b"
format_stamp: "Formatted at 2026-09-28 08:42:10 on dist-test-slave-w6c0"
server_key: "8219c018ca4afda26461ce5eccea4557"
server_key_iv: "aee8a0d8a183f7267475e0c8cc8b25da"
server_key_version: "encryptionkey@0"
I20260928 08:42:10.018875 19505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:10.037297 19505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:10.038205 19505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:10.038607 19505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:10.039680 19505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:42:10.041276 19505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:42:10.041406 19505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:10.041532 19505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:42:10.041628 19505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:10.075294 19505 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42591
I20260928 08:42:10.075353 19631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42591 every 8 connection(s)
I20260928 08:42:10.076678 19505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:42:10.079694 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 19505
I20260928 08:42:10.079881 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:42:10.080595 388 external_mini_cluster.cc:1428] Setting key a833ea32e060d7884e4be474e6c06f7d
I20260928 08:42:10.090066 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:10.079449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60637 (local address 127.0.97.62:36933)
0928 08:42:10.079830 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:10.079837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:10.081543 (+ 1706us) server_negotiation.cc:407] Connection header received
0928 08:42:10.083902 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:10.083906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:10.083974 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:10.084057 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:10.085097 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.085875 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.087521 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.087947 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.088047 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:10.089428 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:10.089433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:10.089436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:10.089456 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:10.089485 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:10.089496 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:10.089561 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:10.089736 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 08:42:10.089819 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260928 08:42:10.091192 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:10.080164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36933 (local address 127.0.97.3:60637)
0928 08:42:10.081354 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:10.081413 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:10.083651 (+ 2238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:10.084275 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:10.084283 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:10.084383 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:10.084960 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:10.084976 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.086015 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.086019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:10.087385 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:10.087407 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.088137 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.088142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:10.088353 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:10.089117 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:42:10.089120 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:10.089163 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:42:10.089250 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:10.089558 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:10.089563 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:10.089568 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:42:10.089705 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 08:42:10.090018 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":340,"threads_started":1}
I20260928 08:42:10.092830 19632 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36933
I20260928 08:42:10.093189 19632 heartbeater.cc:464] Registering TS with master...
I20260928 08:42:10.094089 19632 heartbeater.cc:511] Master 127.0.97.62:36933 requested a full tablet report, sending...
I20260928 08:42:10.095767 19184 ts_manager.cc:194] Registered new tserver with Master: 82cca18ab819496ca9c61446a2baba9b (127.0.97.3:42591)
I20260928 08:42:10.096683 19184 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:60637
I20260928 08:42:10.108413 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:42:10.119029 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:10.112746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52436 (local address 127.0.97.62:36933)
0928 08:42:10.113014 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:10.113021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:10.113353 (+ 332us) server_negotiation.cc:407] Connection header received
0928 08:42:10.113590 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:10.113602 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:10.113678 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:10.113794 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:10.114274 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.115062 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.116401 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.116863 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.117015 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:10.118263 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:10.118269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:10.118272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:10.118290 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:10.118321 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:10.118333 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:10.118481 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:10.118704 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 08:42:10.118774 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 08:42:10.123145 19184 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:42:10.130132 19184 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:42:10.148852 19371 heartbeater.cc:503] Master 127.0.97.62:36933 was elected leader, sending a full tablet report...
I20260928 08:42:10.166219 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:10.143224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40383 (local address 127.0.0.1:52004)
0928 08:42:10.155120 (+ 11896us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:10.155170 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:10.155404 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:10.156304 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:10.156313 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:10.156357 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:10.156967 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:10.156985 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.160173 (+ 3188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.160177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:10.162690 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:10.162706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.165090 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.165097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:10.165381 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:10.165436 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 08:42:10.165579 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 08:42:10.165774 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11603,"mutex_wait_us":2,"thread_start_us":11482,"threads_started":1}
I20260928 08:42:10.166281 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:10.143450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.0.97.1:40383)
0928 08:42:10.154647 (+ 11197us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:10.154670 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:10.155622 (+ 952us) server_negotiation.cc:407] Connection header received
0928 08:42:10.155738 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:10.155746 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:10.156025 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:10.157271 (+ 1246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:10.157672 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.159999 (+ 2327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.162933 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.164508 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.164648 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:10.164945 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:10.165661 (+ 716us) server_negotiation.cc:299] Negotiation successful
0928 08:42:10.165867 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10956,"thread_start_us":10794,"threads_started":1}
I20260928 08:42:10.166796 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:10.143426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34949 (local address 127.0.0.1:50342)
0928 08:42:10.157600 (+ 14174us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:10.157630 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:10.157778 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:10.159376 (+ 1598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:10.159383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:10.159407 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:10.159845 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:10.159861 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.161900 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.161905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:10.164135 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:10.164153 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.166009 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.166015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:10.166230 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:10.166275 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 08:42:10.166342 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 08:42:10.166474 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13998,"thread_start_us":14463,"threads_started":1}
I20260928 08:42:10.168578 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:10.143626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50342 (local address 127.0.97.2:34949)
0928 08:42:10.158768 (+ 15142us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:10.158777 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:10.158807 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:42:10.158900 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:10.158907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:10.159158 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:10.160011 (+ 853us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:10.160260 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.161739 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.164391 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.165802 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.167135 (+ 1333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:10.167367 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:10.167436 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 08:42:10.168233 (+ 797us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14952,"thread_start_us":15999,"threads_started":1}
I20260928 08:42:10.170568 19306 tablet_service.cc:1467] Processing CreateTablet for tablet 393ad262d52544fdad78868b8cf190d1 (DEFAULT_TABLE table=test-table [id=9b4ce72bef6d418bbb0fc56390d8ddb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:42:10.172196 19306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393ad262d52544fdad78868b8cf190d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:10.172384 19437 tablet_service.cc:1467] Processing CreateTablet for tablet 393ad262d52544fdad78868b8cf190d1 (DEFAULT_TABLE table=test-table [id=9b4ce72bef6d418bbb0fc56390d8ddb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:42:10.173542 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:10.160399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42591 (local address 127.0.0.1:38180)
0928 08:42:10.164032 (+ 3633us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:10.164054 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:10.164202 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:10.164976 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:10.164981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:10.165008 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:10.165402 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:10.165410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.167996 (+ 2586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.168004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:10.170340 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:10.170372 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.172751 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.172759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:10.172962 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:10.172999 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 08:42:10.173094 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 08:42:10.173236 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3493,"thread_start_us":3355,"threads_started":1}
I20260928 08:42:10.173741 19437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393ad262d52544fdad78868b8cf190d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:10.174049 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:10.156087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.0.97.3:42591)
0928 08:42:10.163739 (+ 7652us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:10.163752 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:10.164344 (+ 592us) server_negotiation.cc:407] Connection header received
0928 08:42:10.164451 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:10.164459 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:10.164756 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:10.165614 (+ 858us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:10.165901 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.167853 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.170793 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.172538 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.172702 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:10.172982 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:10.173280 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 08:42:10.173462 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7453,"thread_start_us":7287,"threads_started":1}
I20260928 08:42:10.177460 19567 tablet_service.cc:1467] Processing CreateTablet for tablet 393ad262d52544fdad78868b8cf190d1 (DEFAULT_TABLE table=test-table [id=9b4ce72bef6d418bbb0fc56390d8ddb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:42:10.178761 19567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393ad262d52544fdad78868b8cf190d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:10.188529 19650 tablet_bootstrap.cc:492] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2: Bootstrap starting.
I20260928 08:42:10.191413 19651 tablet_bootstrap.cc:492] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182: Bootstrap starting.
I20260928 08:42:10.193912 19650 tablet_bootstrap.cc:654] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:10.194412 19651 tablet_bootstrap.cc:654] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:10.195384 19651 log.cc:824] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:10.195581 19650 log.cc:824] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:10.197783 19652 tablet_bootstrap.cc:492] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b: Bootstrap starting.
I20260928 08:42:10.199833 19651 tablet_bootstrap.cc:492] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182: No bootstrap required, opened a new log
I20260928 08:42:10.200450 19651 ts_tablet_manager.cc:1397] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 08:42:10.201634 19652 tablet_bootstrap.cc:654] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:10.203027 19652 log.cc:824] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:10.205597 19650 tablet_bootstrap.cc:492] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2: No bootstrap required, opened a new log
I20260928 08:42:10.205902 19650 ts_tablet_manager.cc:1397] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.007s
I20260928 08:42:10.209442 19652 tablet_bootstrap.cc:492] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b: No bootstrap required, opened a new log
I20260928 08:42:10.209761 19652 ts_tablet_manager.cc:1397] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260928 08:42:10.210026 19651 raft_consensus.cc:348] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f7ab021ed84316b12fed5f6f04f6b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40383 } } peers { permanent_uuid: "a54227eba6144f448cf857b04000c182" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34949 } } peers { permanent_uuid: "82cca18ab819496ca9c61446a2baba9b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42591 } }
I20260928 08:42:10.210556 19651 raft_consensus.cc:374] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:10.210659 19651 raft_consensus.cc:729] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a54227eba6144f448cf857b04000c182, State: Initialized, Role: FOLLOWER
I20260928 08:42:10.211319 19651 consensus_queue.cc:260] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f7ab021ed84316b12fed5f6f04f6b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40383 } } peers { permanent_uuid: "a54227eba6144f448cf857b04000c182" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34949 } } peers { permanent_uuid: "82cca18ab819496ca9c61446a2baba9b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42591 } }
I20260928 08:42:10.212591 19502 heartbeater.cc:503] Master 127.0.97.62:36933 was elected leader, sending a full tablet report...
I20260928 08:42:10.213482 19651 ts_tablet_manager.cc:1428] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260928 08:42:10.215524 19650 raft_consensus.cc:348] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f7ab021ed84316b12fed5f6f04f6b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40383 } } peers { permanent_uuid: "a54227eba6144f448cf857b04000c182" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34949 } } peers { permanent_uuid: "82cca18ab819496ca9c61446a2baba9b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42591 } }
I20260928 08:42:10.216004 19650 raft_consensus.cc:374] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:10.216125 19650 raft_consensus.cc:729] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f7ab021ed84316b12fed5f6f04f6b2, State: Initialized, Role: FOLLOWER
I20260928 08:42:10.216826 19650 consensus_queue.cc:260] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f7ab021ed84316b12fed5f6f04f6b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40383 } } peers { permanent_uuid: "a54227eba6144f448cf857b04000c182" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34949 } } peers { permanent_uuid: "82cca18ab819496ca9c61446a2baba9b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42591 } }
I20260928 08:42:10.219357 19652 raft_consensus.cc:348] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f7ab021ed84316b12fed5f6f04f6b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40383 } } peers { permanent_uuid: "a54227eba6144f448cf857b04000c182" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34949 } } peers { permanent_uuid: "82cca18ab819496ca9c61446a2baba9b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42591 } }
I20260928 08:42:10.219904 19652 raft_consensus.cc:374] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:10.220012 19652 raft_consensus.cc:729] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82cca18ab819496ca9c61446a2baba9b, State: Initialized, Role: FOLLOWER
I20260928 08:42:10.220786 19650 ts_tablet_manager.cc:1428] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260928 08:42:10.220661 19652 consensus_queue.cc:260] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f7ab021ed84316b12fed5f6f04f6b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40383 } } peers { permanent_uuid: "a54227eba6144f448cf857b04000c182" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34949 } } peers { permanent_uuid: "82cca18ab819496ca9c61446a2baba9b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42591 } }
I20260928 08:42:10.221835 19632 heartbeater.cc:503] Master 127.0.97.62:36933 was elected leader, sending a full tablet report...
I20260928 08:42:10.222285 19652 ts_tablet_manager.cc:1428] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
W20260928 08:42:10.328459 19633 tablet.cc:2369] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:42:10.365139 19503 tablet.cc:2369] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:42:10.368304 19372 tablet.cc:2369] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:42:10.380545 19658 raft_consensus.cc:482] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:42:10.380918 19658 raft_consensus.cc:504] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f7ab021ed84316b12fed5f6f04f6b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40383 } } peers { permanent_uuid: "a54227eba6144f448cf857b04000c182" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34949 } } peers { permanent_uuid: "82cca18ab819496ca9c61446a2baba9b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42591 } }
I20260928 08:42:10.382834 19658 leader_election.cc:288] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14f7ab021ed84316b12fed5f6f04f6b2 (127.0.97.1:40383), a54227eba6144f448cf857b04000c182 (127.0.97.2:34949)
I20260928 08:42:10.388893 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:10.383321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54863 (local address 127.0.97.1:40383)
0928 08:42:10.383535 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:10.383544 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:10.383830 (+ 286us) server_negotiation.cc:407] Connection header received
0928 08:42:10.384073 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:10.384078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:10.384152 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:10.384281 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:10.384682 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.385388 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.387215 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.387985 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.388124 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:10.388236 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:10.388492 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 08:42:10.388633 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:42:10.388947 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:10.383276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40383 (local address 127.0.97.3:54863)
0928 08:42:10.383688 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:10.383712 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:10.383889 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:10.384305 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:10.384310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:10.384337 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:10.384527 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:10.384536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.385547 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.385554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:10.387056 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:10.387068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.388162 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.388168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:10.388335 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:10.388360 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 08:42:10.388464 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 08:42:10.388610 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":1}
I20260928 08:42:10.389504 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:10.383293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58377 (local address 127.0.97.2:34949)
0928 08:42:10.383606 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:10.383614 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:10.384239 (+ 625us) server_negotiation.cc:407] Connection header received
0928 08:42:10.384451 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:10.384457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:10.384537 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:10.384648 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:10.385229 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.385964 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.387804 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.388548 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.388708 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:10.388798 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:10.389144 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 08:42:10.389276 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260928 08:42:10.390082 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:10.383320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34949 (local address 127.0.97.3:58377)
0928 08:42:10.384135 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:10.384152 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:10.384313 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:10.384659 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:10.384663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:10.384685 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:10.385076 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:10.385085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.386151 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.386156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:10.387619 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:10.387630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.388727 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.388733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:10.388910 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:10.388937 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 08:42:10.389713 (+ 776us) client_negotiation.cc:240] Negotiation successful
0928 08:42:10.389844 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":23,"thread_start_us":457,"threads_started":1}
I20260928 08:42:10.390050 19326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393ad262d52544fdad78868b8cf190d1" candidate_uuid: "82cca18ab819496ca9c61446a2baba9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f7ab021ed84316b12fed5f6f04f6b2" is_pre_election: true
I20260928 08:42:10.390710 19326 raft_consensus.cc:2454] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82cca18ab819496ca9c61446a2baba9b in term 0.
I20260928 08:42:10.391211 19457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393ad262d52544fdad78868b8cf190d1" candidate_uuid: "82cca18ab819496ca9c61446a2baba9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a54227eba6144f448cf857b04000c182" is_pre_election: true
I20260928 08:42:10.391538 19522 leader_election.cc:302] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14f7ab021ed84316b12fed5f6f04f6b2, 82cca18ab819496ca9c61446a2baba9b; no voters:
I20260928 08:42:10.391776 19457 raft_consensus.cc:2454] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82cca18ab819496ca9c61446a2baba9b in term 0.
I20260928 08:42:10.392014 19658 raft_consensus.cc:2781] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:42:10.392169 19658 raft_consensus.cc:482] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:42:10.392290 19658 raft_consensus.cc:3037] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:10.394680 19658 raft_consensus.cc:504] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f7ab021ed84316b12fed5f6f04f6b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40383 } } peers { permanent_uuid: "a54227eba6144f448cf857b04000c182" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34949 } } peers { permanent_uuid: "82cca18ab819496ca9c61446a2baba9b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42591 } }
I20260928 08:42:10.395292 19658 leader_election.cc:288] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [CANDIDATE]: Term 1 election: Requested vote from peers 14f7ab021ed84316b12fed5f6f04f6b2 (127.0.97.1:40383), a54227eba6144f448cf857b04000c182 (127.0.97.2:34949)
I20260928 08:42:10.395818 19326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393ad262d52544fdad78868b8cf190d1" candidate_uuid: "82cca18ab819496ca9c61446a2baba9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f7ab021ed84316b12fed5f6f04f6b2"
I20260928 08:42:10.395844 19457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393ad262d52544fdad78868b8cf190d1" candidate_uuid: "82cca18ab819496ca9c61446a2baba9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a54227eba6144f448cf857b04000c182"
I20260928 08:42:10.396062 19457 raft_consensus.cc:3037] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:10.396109 19326 raft_consensus.cc:3037] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:10.398485 19326 raft_consensus.cc:2454] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cca18ab819496ca9c61446a2baba9b in term 1.
I20260928 08:42:10.398592 19457 raft_consensus.cc:2454] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82cca18ab819496ca9c61446a2baba9b in term 1.
I20260928 08:42:10.399097 19522 leader_election.cc:302] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14f7ab021ed84316b12fed5f6f04f6b2, 82cca18ab819496ca9c61446a2baba9b; no voters:
I20260928 08:42:10.399438 19658 raft_consensus.cc:2781] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:10.399919 19658 raft_consensus.cc:686] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [term 1 LEADER]: Becoming Leader. State: Replica: 82cca18ab819496ca9c61446a2baba9b, State: Running, Role: LEADER
I20260928 08:42:10.400421 19658 consensus_queue.cc:237] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f7ab021ed84316b12fed5f6f04f6b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40383 } } peers { permanent_uuid: "a54227eba6144f448cf857b04000c182" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34949 } } peers { permanent_uuid: "82cca18ab819496ca9c61446a2baba9b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42591 } }
I20260928 08:42:10.405431 19185 catalog_manager.cc:5495] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b reported cstate change: term changed from 0 to 1, leader changed from <none> to 82cca18ab819496ca9c61446a2baba9b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "82cca18ab819496ca9c61446a2baba9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f7ab021ed84316b12fed5f6f04f6b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a54227eba6144f448cf857b04000c182" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82cca18ab819496ca9c61446a2baba9b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42591 } health_report { overall_health: HEALTHY } } }
I20260928 08:42:10.475903 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:10.470700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38194 (local address 127.0.97.3:42591)
0928 08:42:10.470929 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:10.470938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:10.470991 (+ 53us) server_negotiation.cc:407] Connection header received
0928 08:42:10.471190 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:10.471194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:10.471283 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:10.471408 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:42:10.471717 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.472426 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.473742 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.474148 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.474237 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:10.474783 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:42:10.475298 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:42:10.475428 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:10.475535 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 08:42:10.475609 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:42:10.484546 19457 raft_consensus.cc:1261] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182 [term 1 FOLLOWER]: Refusing update from remote peer 82cca18ab819496ca9c61446a2baba9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:42:10.484546 19326 raft_consensus.cc:1261] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2 [term 1 FOLLOWER]: Refusing update from remote peer 82cca18ab819496ca9c61446a2baba9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:42:10.485707 19660 consensus_queue.cc:1035] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f7ab021ed84316b12fed5f6f04f6b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:42:10.486117 19658 consensus_queue.cc:1035] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a54227eba6144f448cf857b04000c182" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:42:10.498757 19664 mvcc.cc:205] Tried to move back new op lower bound from 7334235875254202368 to 7334235874925015040. Current Snapshot: MvccSnapshot[applied={T|T < 7334235875254202368}]
I20260928 08:42:10.499751 19663 mvcc.cc:205] Tried to move back new op lower bound from 7334235875254202368 to 7334235874925015040. Current Snapshot: MvccSnapshot[applied={T|T < 7334235875254202368}]
I20260928 08:42:10.503062 19665 mvcc.cc:205] Tried to move back new op lower bound from 7334235875254202368 to 7334235874925015040. Current Snapshot: MvccSnapshot[applied={T|T < 7334235875254202368}]
I20260928 08:42:10.515766 19184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:42:10.516052 19184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:42:10.519623 19184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 232d6c7405ca41f6a34ed7d921219d4c: Sending DeleteTablet for 3 replicas of tablet 393ad262d52544fdad78868b8cf190d1
I20260928 08:42:10.520884 19567 tablet_service.cc:1514] Processing DeleteTablet for tablet 393ad262d52544fdad78868b8cf190d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:42:10 UTC) from {username='slave'} at 127.0.0.1:38180
I20260928 08:42:10.520874 19437 tablet_service.cc:1514] Processing DeleteTablet for tablet 393ad262d52544fdad78868b8cf190d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:42:10 UTC) from {username='slave'} at 127.0.0.1:50342
I20260928 08:42:10.521229 19306 tablet_service.cc:1514] Processing DeleteTablet for tablet 393ad262d52544fdad78868b8cf190d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:42:10 UTC) from {username='slave'} at 127.0.0.1:52004
I20260928 08:42:10.522927 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 19244
I20260928 08:42:10.533960 19674 tablet_replica.cc:330] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2: stopping tablet replica
I20260928 08:42:10.534616 19674 raft_consensus.cc:2229] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:42:10.535303 19674 raft_consensus.cc:2258] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:42:10.536166 19672 tablet_replica.cc:330] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b: stopping tablet replica
I20260928 08:42:10.536609 19672 raft_consensus.cc:2229] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:42:10.537410 19672 raft_consensus.cc:2258] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:42:10.539911 19672 ts_tablet_manager.cc:1905] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:42:10.540122 19674 ts_tablet_manager.cc:1905] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:42:10.543579 19673 tablet_replica.cc:330] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182: stopping tablet replica
I20260928 08:42:10.544065 19673 raft_consensus.cc:2229] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:42:10.544595 19673 raft_consensus.cc:2258] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:42:10.546957 19673 ts_tablet_manager.cc:1905] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:42:10.548024 19672 ts_tablet_manager.cc:1918] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:42:10.548214 19672 log.cc:1196] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/393ad262d52544fdad78868b8cf190d1
I20260928 08:42:10.548774 19672 ts_tablet_manager.cc:1939] T 393ad262d52544fdad78868b8cf190d1 P 82cca18ab819496ca9c61446a2baba9b: Deleting consensus metadata
I20260928 08:42:10.550058 19674 ts_tablet_manager.cc:1918] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:42:10.550221 19674 log.cc:1196] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-0/wal/wals/393ad262d52544fdad78868b8cf190d1
I20260928 08:42:10.550799 19674 ts_tablet_manager.cc:1939] T 393ad262d52544fdad78868b8cf190d1 P 14f7ab021ed84316b12fed5f6f04f6b2: Deleting consensus metadata
I20260928 08:42:10.550783 19171 catalog_manager.cc:4841] TS 82cca18ab819496ca9c61446a2baba9b (127.0.97.3:42591): tablet 393ad262d52544fdad78868b8cf190d1 (table test-table [id=9b4ce72bef6d418bbb0fc56390d8ddb5]) successfully deleted
I20260928 08:42:10.552523 19171 catalog_manager.cc:4841] TS 14f7ab021ed84316b12fed5f6f04f6b2 (127.0.97.1:40383): tablet 393ad262d52544fdad78868b8cf190d1 (table test-table [id=9b4ce72bef6d418bbb0fc56390d8ddb5]) successfully deleted
I20260928 08:42:10.553478 19673 ts_tablet_manager.cc:1918] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:42:10.553642 19673 log.cc:1196] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/393ad262d52544fdad78868b8cf190d1
I20260928 08:42:10.554092 19673 ts_tablet_manager.cc:1939] T 393ad262d52544fdad78868b8cf190d1 P a54227eba6144f448cf857b04000c182: Deleting consensus metadata
I20260928 08:42:10.555599 19169 catalog_manager.cc:4841] TS a54227eba6144f448cf857b04000c182 (127.0.97.2:34949): tablet 393ad262d52544fdad78868b8cf190d1 (table test-table [id=9b4ce72bef6d418bbb0fc56390d8ddb5]) successfully deleted
I20260928 08:42:10.564371 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:10.524371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52010 (local address 127.0.97.1:40383)
0928 08:42:10.524536 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:10.524543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:10.539090 (+ 14547us) server_negotiation.cc:407] Connection header received
0928 08:42:10.539631 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:10.539639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:10.539747 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:10.539830 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:10.540480 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.541692 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.543850 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.544468 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.544576 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:10.562722 (+ 18146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:10.562739 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:10.562762 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:10.563457 (+ 695us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:10.563511 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:10.563535 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:10.563715 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:10.563860 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 08:42:10.563983 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 08:42:10.565086 19366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:10.670195 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 19244
I20260928 08:42:10.695026 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 19375
I20260928 08:42:10.702798 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:10.695935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50358 (local address 127.0.97.2:34949)
0928 08:42:10.696139 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:10.696144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:10.696166 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:42:10.696237 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:10.696242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:10.696335 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:10.696446 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:10.696891 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.697736 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.699340 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.699882 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.700045 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:10.701264 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:10.701273 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:10.701287 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:10.701988 (+ 701us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:10.702039 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:10.702064 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:10.702194 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:10.702329 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 08:42:10.702444 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 08:42:10.703575 19497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:10.795670 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 19375
I20260928 08:42:10.820570 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 19505
I20260928 08:42:10.828615 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:10.821394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38196 (local address 127.0.97.3:42591)
0928 08:42:10.821570 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:10.821575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:10.821632 (+ 57us) server_negotiation.cc:407] Connection header received
0928 08:42:10.821821 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:10.821825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:10.821924 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:10.822021 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:10.822538 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.823473 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.825337 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:10.825769 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:10.825864 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:10.827141 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:10.827150 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:10.827165 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:10.827788 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:10.827837 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:10.827860 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:10.827943 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:10.828141 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 08:42:10.828228 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:42:10.829473 19627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:10.952759 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 19505
I20260928 08:42:10.972232 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 19154
I20260928 08:42:10.973351 19215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:11.084131 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 19154
2026-09-28T08:42:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3039 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:11 dist-test-slave-w6c0 krb5kdc[19682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:11 dist-test-slave-w6c0 krb5kdc[19682](info): set up 2 sockets
Sep 28 08:42:11 dist-test-slave-w6c0 krb5kdc[19682](info): commencing operation
krb5kdc: starting...
W20260928 08:42:12.190842 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:12 dist-test-slave-w6c0 krb5kdc[19682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:42:12.434362 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40437
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:12.765868 19695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:12.766204 19695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:12.766359 19695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:12.773635 19695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:12.773777 19695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:12.773880 19695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:12.773977 19695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:12.774062 19695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:12.783980 19695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:12.785367 19695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:12.786715 19695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:12.798209 19700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:12.798249 19701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:12.798779 19703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:12.799211 19695 server_base.cc:1034] running on GCE node
I20260928 08:42:12.799985 19695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:12.801330 19695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:12.802590 19695 hybrid_clock.cc:648] HybridClock initialized: now 1790584932802550 us; error 44 us; skew 500 ppm
I20260928 08:42:12.802922 19695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:12 dist-test-slave-w6c0 krb5kdc[19682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:12.805892 19695 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:42:12.807322 19695 webserver.cc:460] Webserver started at http://127.0.97.62:35753/ using document root <none> and password file <none>
I20260928 08:42:12.808009 19695 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:12.808143 19695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:12.808442 19695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:12.811378 19695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "0646c471056c4db79e42830b8ce3f69c"
format_stamp: "Formatted at 2026-09-28 08:42:12 on dist-test-slave-w6c0"
server_key: "7cb7dcf79edae700d4c2de4fb8f7e84b"
server_key_iv: "7332f0ad44d62d5a918d8e55446f0fea"
server_key_version: "encryptionkey@0"
I20260928 08:42:12.812117 19695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "0646c471056c4db79e42830b8ce3f69c"
format_stamp: "Formatted at 2026-09-28 08:42:12 on dist-test-slave-w6c0"
server_key: "7cb7dcf79edae700d4c2de4fb8f7e84b"
server_key_iv: "7332f0ad44d62d5a918d8e55446f0fea"
server_key_version: "encryptionkey@0"
I20260928 08:42:12.817039 19695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:42:12.820307 19710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:12.821491 19695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 08:42:12.821664 19695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "0646c471056c4db79e42830b8ce3f69c"
format_stamp: "Formatted at 2026-09-28 08:42:12 on dist-test-slave-w6c0"
server_key: "7cb7dcf79edae700d4c2de4fb8f7e84b"
server_key_iv: "7332f0ad44d62d5a918d8e55446f0fea"
server_key_version: "encryptionkey@0"
I20260928 08:42:12.821851 19695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:12.842916 19695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:12.846735 19695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:12.846995 19695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:12.865052 19695 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40437
I20260928 08:42:12.865085 19761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40437 every 8 connection(s)
I20260928 08:42:12.866416 19695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:12.867967 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 19695
I20260928 08:42:12.868222 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:12.868657 388 external_mini_cluster.cc:1428] Setting key 569df6ddb4f0cd2afee8f46592ddc261
I20260928 08:42:12.870908 19762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 08:42:12 dist-test-slave-w6c0 krb5kdc[19682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:42:12.883554 19762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c: Bootstrap starting.
I20260928 08:42:12.887537 19762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:12.888749 19762 log.cc:824] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:12.891978 19762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c: No bootstrap required, opened a new log
I20260928 08:42:12.892437 19765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:12.870401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34030 (local address 127.0.97.62:40437)
0928 08:42:12.871240 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:12.871255 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:12.871304 (+ 49us) server_negotiation.cc:407] Connection header received
0928 08:42:12.873577 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:12.873603 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:12.874077 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:12.874561 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:12.875866 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:12.877889 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:12.879189 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:12.880191 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:12.880384 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:12.884317 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:12.884327 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:12.884344 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:12.884384 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:12.888223 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:12.888896 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:12.888904 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:12.888911 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:12.889107 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:12.889545 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:12.889549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:12.889551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:12.890123 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:12.890423 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:12.890781 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 08:42:12.891191 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":332,"threads_started":1}
I20260928 08:42:12.899973 19762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0646c471056c4db79e42830b8ce3f69c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } }
I20260928 08:42:12.900395 19762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:12.900480 19762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0646c471056c4db79e42830b8ce3f69c, State: Initialized, Role: FOLLOWER
I20260928 08:42:12.901065 19762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0646c471056c4db79e42830b8ce3f69c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } }
I20260928 08:42:12.901281 19762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:12.901434 19762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:12.901590 19762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:12.903748 19762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0646c471056c4db79e42830b8ce3f69c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } }
I20260928 08:42:12.904299 19762 leader_election.cc:302] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0646c471056c4db79e42830b8ce3f69c; no voters:
I20260928 08:42:12.904879 19762 leader_election.cc:288] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:12.905062 19767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:12.905725 19767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c [term 1 LEADER]: Becoming Leader. State: Replica: 0646c471056c4db79e42830b8ce3f69c, State: Running, Role: LEADER
I20260928 08:42:12.906221 19767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0646c471056c4db79e42830b8ce3f69c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } }
I20260928 08:42:12.906988 19762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:12.909090 19769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0646c471056c4db79e42830b8ce3f69c. Latest consensus state: current_term: 1 leader_uuid: "0646c471056c4db79e42830b8ce3f69c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0646c471056c4db79e42830b8ce3f69c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } }
I20260928 08:42:12.909057 19768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0646c471056c4db79e42830b8ce3f69c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0646c471056c4db79e42830b8ce3f69c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } }
I20260928 08:42:12.909528 19769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:12.909514 19768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:12.910557 19774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:12.917325 19774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:12.927211 19774 catalog_manager.cc:1347] Generated new cluster ID: 2001e32c2c36424aa8657511d872b15f
I20260928 08:42:12.927320 19774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:12.939651 19774 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:12.940924 19774 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:12.949373 19774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c: Generated new TSK 0
I20260928 08:42:12.950119 19774 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:42:13.013258 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40437
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:13.346757 19788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:13.347035 19788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:13.347131 19788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:13.354316 19788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:13.354436 19788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:13.354610 19788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:42:13.364291 19788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:13.365645 19788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:13.367048 19788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:13.380584 19793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:13.380662 19794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:13.380801 19796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:13.380688 19788 server_base.cc:1034] running on GCE node
I20260928 08:42:13.381614 19788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:13.382468 19788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:13.383726 19788 hybrid_clock.cc:648] HybridClock initialized: now 1790584933383684 us; error 37 us; skew 500 ppm
I20260928 08:42:13.384044 19788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:13 dist-test-slave-w6c0 krb5kdc[19682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:13.387125 19788 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:42:13.388597 19788 webserver.cc:460] Webserver started at http://127.0.97.1:37333/ using document root <none> and password file <none>
I20260928 08:42:13.389283 19788 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:13.389420 19788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:13.389719 19788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:13.392578 19788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "35b3015bc34e4062b00c4b2732b932b9"
format_stamp: "Formatted at 2026-09-28 08:42:13 on dist-test-slave-w6c0"
server_key: "d94161a839bfd0809de4e76283f321f6"
server_key_iv: "6965de7f65a7216d552d2184e16aaa2e"
server_key_version: "encryptionkey@0"
I20260928 08:42:13.393332 19788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "35b3015bc34e4062b00c4b2732b932b9"
format_stamp: "Formatted at 2026-09-28 08:42:13 on dist-test-slave-w6c0"
server_key: "d94161a839bfd0809de4e76283f321f6"
server_key_iv: "6965de7f65a7216d552d2184e16aaa2e"
server_key_version: "encryptionkey@0"
I20260928 08:42:13.398314 19788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260928 08:42:13.401700 19803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:13.402990 19788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:42:13.403149 19788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "35b3015bc34e4062b00c4b2732b932b9"
format_stamp: "Formatted at 2026-09-28 08:42:13 on dist-test-slave-w6c0"
server_key: "d94161a839bfd0809de4e76283f321f6"
server_key_iv: "6965de7f65a7216d552d2184e16aaa2e"
server_key_version: "encryptionkey@0"
I20260928 08:42:13.403342 19788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:13.431885 19788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:13.435807 19788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:13.436108 19788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:13.437080 19788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:42:13.438378 19788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:42:13.438481 19788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:13.438629 19788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:42:13.438709 19788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:13.471601 19788 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45041
I20260928 08:42:13.471681 19915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45041 every 8 connection(s)
I20260928 08:42:13.472852 19788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:42:13.477694 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 19788
I20260928 08:42:13.477872 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:42:13.478272 388 external_mini_cluster.cc:1428] Setting key f36b4b821395faaab7cecd48a9d90bdc
Sep 28 08:42:13 dist-test-slave-w6c0 krb5kdc[19682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:42:13.493480 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:13.475323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57681 (local address 127.0.97.62:40437)
0928 08:42:13.475808 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:13.475814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:13.476664 (+ 850us) server_negotiation.cc:407] Connection header received
0928 08:42:13.478638 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:13.478644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:13.478752 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:13.478881 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:13.480465 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:13.481573 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:13.483192 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:13.483880 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:13.484031 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:13.487380 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:13.487389 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:13.487397 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:13.487441 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:13.490276 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:13.491212 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:13.491216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:13.491219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:13.491334 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:13.491802 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:13.491805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:13.491807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:13.492057 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:13.492168 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:13.493048 (+ 880us) server_negotiation.cc:299] Negotiation successful
0928 08:42:13.493215 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":233,"threads_started":1}
I20260928 08:42:13.495019 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:13.475819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.1:57681)
0928 08:42:13.476516 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:13.476554 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:13.478321 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:13.479140 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:13.479156 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:13.479634 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:13.480299 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:13.480320 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:13.481732 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:13.481736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:13.483051 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:13.483064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:13.484086 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:13.484091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:13.484282 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:13.484959 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:42:13.484982 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:13.487150 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:13.490419 (+ 3269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:13.490431 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:13.490449 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:13.491095 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:13.491436 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:13.491444 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:13.491450 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:13.491697 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:13.492179 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:13.492189 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:13.492661 (+ 472us) client_negotiation.cc:769] Sending connection context
0928 08:42:13.492951 (+ 290us) client_negotiation.cc:240] Negotiation successful
0928 08:42:13.493544 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":279,"threads_started":1}
I20260928 08:42:13.496655 19916 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40437
I20260928 08:42:13.496954 19916 heartbeater.cc:464] Registering TS with master...
I20260928 08:42:13.497663 19916 heartbeater.cc:511] Master 127.0.97.62:40437 requested a full tablet report, sending...
I20260928 08:42:13.499979 19727 ts_manager.cc:194] Registered new tserver with Master: 35b3015bc34e4062b00c4b2732b932b9 (127.0.97.1:45041)
I20260928 08:42:13.502164 19727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57681
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:42:13.537153 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40437
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:13.856346 19922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:13.856618 19922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:13.856742 19922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:13.864037 19922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:13.864184 19922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:13.864352 19922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:42:13.879741 19922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:13.881279 19922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:13.882939 19922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:13.897219 19927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:13.897226 19928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:13.897171 19930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:13.897854 19922 server_base.cc:1034] running on GCE node
I20260928 08:42:13.898561 19922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:13.899601 19922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:13.900826 19922 hybrid_clock.cc:648] HybridClock initialized: now 1790584933900749 us; error 57 us; skew 500 ppm
I20260928 08:42:13.901168 19922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:13 dist-test-slave-w6c0 krb5kdc[19682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:13.904353 19922 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:42:13.905828 19922 webserver.cc:460] Webserver started at http://127.0.97.2:38371/ using document root <none> and password file <none>
I20260928 08:42:13.906576 19922 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:13.906713 19922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:13.907095 19922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:13.910187 19922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "8400ba9e115243acaedbac0396c2dbad"
format_stamp: "Formatted at 2026-09-28 08:42:13 on dist-test-slave-w6c0"
server_key: "194baf9dea1eb5b6d80dcc3531faae89"
server_key_iv: "376dec86dcc857e2ab05a308ff4148d0"
server_key_version: "encryptionkey@0"
I20260928 08:42:13.911018 19922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8400ba9e115243acaedbac0396c2dbad"
format_stamp: "Formatted at 2026-09-28 08:42:13 on dist-test-slave-w6c0"
server_key: "194baf9dea1eb5b6d80dcc3531faae89"
server_key_iv: "376dec86dcc857e2ab05a308ff4148d0"
server_key_version: "encryptionkey@0"
I20260928 08:42:13.916105 19922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 08:42:13.919400 19937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:13.920686 19922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:42:13.920858 19922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "8400ba9e115243acaedbac0396c2dbad"
format_stamp: "Formatted at 2026-09-28 08:42:13 on dist-test-slave-w6c0"
server_key: "194baf9dea1eb5b6d80dcc3531faae89"
server_key_iv: "376dec86dcc857e2ab05a308ff4148d0"
server_key_version: "encryptionkey@0"
I20260928 08:42:13.921054 19922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:13.937842 19922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:13.942009 19922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:13.942294 19922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:13.943248 19922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:42:13.944530 19922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:42:13.944628 19922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:13.944796 19922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:42:13.944865 19922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:13.977097 19922 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43337
I20260928 08:42:13.977187 20049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43337 every 8 connection(s)
I20260928 08:42:13.978358 19922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:42:13.980640 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 19922
I20260928 08:42:13.980840 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:42:13.981287 388 external_mini_cluster.cc:1428] Setting key 336185b7c0349f9cf227e61f1bd084a3
Sep 28 08:42:13 dist-test-slave-w6c0 krb5kdc[19682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:42:13.998968 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:13.980943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44515 (local address 127.0.97.62:40437)
0928 08:42:13.981148 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:13.981154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:13.982410 (+ 1256us) server_negotiation.cc:407] Connection header received
0928 08:42:13.984259 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:13.984267 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:13.984360 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:13.984472 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:13.985861 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:13.987005 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:13.988525 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:13.989271 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:13.989411 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:13.993121 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:13.993127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:13.993137 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:13.993170 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:13.996015 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:13.996711 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:13.996717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:13.996720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:13.996848 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:13.997266 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:13.997272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:13.997276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:13.997533 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:13.997686 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:13.998356 (+ 670us) server_negotiation.cc:299] Negotiation successful
0928 08:42:13.998616 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:42:13.999964 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:13.981448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.2:44515)
0928 08:42:13.982241 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:13.982298 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:13.983990 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:13.984661 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:13.984668 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:13.985153 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:13.985693 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:13.985709 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:13.987149 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:13.987152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:13.988382 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:13.988393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:13.989464 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:13.989469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:13.989664 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:13.990348 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:42:13.990371 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:13.992894 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:13.996146 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:13.996155 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:13.996173 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:13.996585 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:13.996956 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:13.996962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:13.996965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:13.997157 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:13.997658 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:13.997666 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:13.998008 (+ 342us) client_negotiation.cc:769] Sending connection context
0928 08:42:13.998281 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 08:42:13.998852 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":334,"threads_started":1}
I20260928 08:42:14.001453 20050 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40437
I20260928 08:42:14.001753 20050 heartbeater.cc:464] Registering TS with master...
I20260928 08:42:14.002444 20050 heartbeater.cc:511] Master 127.0.97.62:40437 requested a full tablet report, sending...
I20260928 08:42:14.003953 19727 ts_manager.cc:194] Registered new tserver with Master: 8400ba9e115243acaedbac0396c2dbad (127.0.97.2:43337)
I20260928 08:42:14.004747 19727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44515
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:42:14.036063 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40437
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:14.364557 20055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:14.364851 20055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:14.364946 20055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:14.372167 20055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:14.372280 20055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:14.372392 20055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:42:14.382277 20055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:14.383638 20055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:14.384907 20055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:14.398569 20060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:14.398640 20063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:14.398667 20061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:14.399618 20055 server_base.cc:1034] running on GCE node
I20260928 08:42:14.400219 20055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:14.401167 20055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:14.402384 20055 hybrid_clock.cc:648] HybridClock initialized: now 1790584934402333 us; error 45 us; skew 500 ppm
I20260928 08:42:14.402814 20055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:14 dist-test-slave-w6c0 krb5kdc[19682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:14.405894 20055 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:42:14.407510 20055 webserver.cc:460] Webserver started at http://127.0.97.3:35321/ using document root <none> and password file <none>
I20260928 08:42:14.408362 20055 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:14.408463 20055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:14.408726 20055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:14.411712 20055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "43c51f97676b418398b9947560bbab22"
format_stamp: "Formatted at 2026-09-28 08:42:14 on dist-test-slave-w6c0"
server_key: "9b825a213ee044c0e853a367a21f0a18"
server_key_iv: "e991c65b03e30bcc2a993797a28969e9"
server_key_version: "encryptionkey@0"
I20260928 08:42:14.412439 20055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "43c51f97676b418398b9947560bbab22"
format_stamp: "Formatted at 2026-09-28 08:42:14 on dist-test-slave-w6c0"
server_key: "9b825a213ee044c0e853a367a21f0a18"
server_key_iv: "e991c65b03e30bcc2a993797a28969e9"
server_key_version: "encryptionkey@0"
I20260928 08:42:14.417371 20055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:42:14.420791 20070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:14.422125 20055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 08:42:14.422258 20055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "43c51f97676b418398b9947560bbab22"
format_stamp: "Formatted at 2026-09-28 08:42:14 on dist-test-slave-w6c0"
server_key: "9b825a213ee044c0e853a367a21f0a18"
server_key_iv: "e991c65b03e30bcc2a993797a28969e9"
server_key_version: "encryptionkey@0"
I20260928 08:42:14.422416 20055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:14.445541 20055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:14.449589 20055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:14.449843 20055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:14.450863 20055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:42:14.452119 20055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:42:14.452191 20055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:14.452277 20055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:42:14.452322 20055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:14.485242 20055 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40833
I20260928 08:42:14.485355 20182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40833 every 8 connection(s)
I20260928 08:42:14.486545 20055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:42:14.491209 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 20055
I20260928 08:42:14.491393 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:42:14.491994 388 external_mini_cluster.cc:1428] Setting key b1a8700b14ca6eeac279894d88352032
Sep 28 08:42:14 dist-test-slave-w6c0 krb5kdc[19682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:42:14.505287 19916 heartbeater.cc:503] Master 127.0.97.62:40437 was elected leader, sending a full tablet report...
I20260928 08:42:14.507895 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:14.489579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45617 (local address 127.0.97.62:40437)
0928 08:42:14.489795 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:14.489799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:14.490942 (+ 1143us) server_negotiation.cc:407] Connection header received
0928 08:42:14.493338 (+ 2396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:14.493345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:14.493424 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:14.493532 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:14.495328 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.496108 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.497577 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.497959 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.498055 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:14.501397 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:14.501402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:14.501405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:14.501447 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:14.504172 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:14.504882 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:14.504886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:14.504888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:14.505013 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:14.505577 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:14.505583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:14.505589 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:14.505981 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:14.506196 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:14.507220 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0928 08:42:14.507534 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 08:42:14.508930 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:14.489994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.3:45617)
0928 08:42:14.490773 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:14.490815 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:14.492929 (+ 2114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:14.493746 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:14.493754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:14.494472 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:14.495179 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:14.495196 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.496241 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.496244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:14.497445 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:14.497456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.498157 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.498162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:14.498348 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:14.499036 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:42:14.499057 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:14.501200 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:14.504311 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:14.504318 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:14.504331 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:14.504774 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:14.505153 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:14.505158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:14.505161 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:14.505398 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:14.506251 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:14.506258 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:14.506727 (+ 469us) client_negotiation.cc:769] Sending connection context
0928 08:42:14.507050 (+ 323us) client_negotiation.cc:240] Negotiation successful
0928 08:42:14.507680 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":279,"threads_started":1}
I20260928 08:42:14.510334 20183 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40437
I20260928 08:42:14.510681 20183 heartbeater.cc:464] Registering TS with master...
I20260928 08:42:14.511404 20183 heartbeater.cc:511] Master 127.0.97.62:40437 requested a full tablet report, sending...
I20260928 08:42:14.512871 19726 ts_manager.cc:194] Registered new tserver with Master: 43c51f97676b418398b9947560bbab22 (127.0.97.3:40833)
I20260928 08:42:14.513760 19726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45617
I20260928 08:42:14.519855 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:42:14.537710 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:14.524146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35468 (local address 127.0.97.62:40437)
0928 08:42:14.524311 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:14.524316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:14.524853 (+ 537us) server_negotiation.cc:407] Connection header received
0928 08:42:14.525101 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:14.525105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:14.525208 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:14.525347 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:14.526986 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.527888 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.529161 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.529592 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.529697 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:14.532196 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:14.532201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:14.532205 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:14.532251 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:14.535200 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:14.535840 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:14.535844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:14.535847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:14.535972 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:14.536287 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:14.536298 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:14.536301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:14.536679 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:14.536879 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:14.537242 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 08:42:14.537440 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 08:42:14.541927 19726 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:42:14.549118 19726 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:42:14.571380 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:14.561210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40833 (local address 127.0.0.1:43010)
0928 08:42:14.563281 (+ 2071us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:14.563302 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:14.563490 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:14.564473 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:14.564478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:14.564515 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:14.564970 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:14.564981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.566525 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.566530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:14.568684 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:14.568695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.570273 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.570280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:14.570477 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:14.570554 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 08:42:14.570699 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 08:42:14.570895 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1936,"mutex_wait_us":41,"thread_start_us":872,"threads_started":1}
I20260928 08:42:14.571623 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:14.561787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.0.97.3:40833)
0928 08:42:14.562841 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:14.562851 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:14.563390 (+ 539us) server_negotiation.cc:407] Connection header received
0928 08:42:14.563667 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:14.563674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:14.563929 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:14.564121 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:14.565189 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.566364 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.568833 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.570079 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.570232 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:14.570470 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:14.570759 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 08:42:14.571275 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":880,"thread_start_us":864,"threads_started":1}
I20260928 08:42:14.571981 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:14.561064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.0.1:39678)
0928 08:42:14.561965 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:14.562005 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:14.562184 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:14.564013 (+ 1829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:14.564020 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:14.564050 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:14.564599 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:14.564614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.566807 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.566812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:14.569224 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:14.569238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.571073 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.571081 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:14.571271 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:14.571303 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 08:42:14.571376 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 08:42:14.571514 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":54,"thread_start_us":1933,"threads_started":1}
I20260928 08:42:14.572803 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:14.563638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.0.97.2:43337)
0928 08:42:14.564182 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:14.564191 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:14.564756 (+ 565us) server_negotiation.cc:407] Connection header received
0928 08:42:14.565063 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:14.565071 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:14.565303 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:14.565461 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:14.566292 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.567897 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.570028 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.571454 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.571590 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:14.571767 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:14.572362 (+ 595us) server_negotiation.cc:299] Negotiation successful
0928 08:42:14.572524 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":246,"threads_started":1}
I20260928 08:42:14.573292 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:14.561275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39678 (local address 127.0.97.1:45041)
0928 08:42:14.563344 (+ 2069us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:14.563354 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:14.563384 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:42:14.563489 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:14.563497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:14.563740 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:14.564824 (+ 1084us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:14.565053 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.566666 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.569389 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.570828 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.572367 (+ 1539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:14.572598 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:14.572679 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 08:42:14.572853 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1896,"thread_start_us":1728,"threads_started":1}
I20260928 08:42:14.573407 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:14.563414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43337 (local address 127.0.0.1:60534)
0928 08:42:14.564642 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:14.564672 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:14.564861 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:14.565726 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:14.565730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:14.565754 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:14.566088 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:14.566100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.568069 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.568075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:14.569885 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:14.569896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.571938 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.571946 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:14.572138 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:14.572183 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 08:42:14.572914 (+ 731us) client_negotiation.cc:240] Negotiation successful
0928 08:42:14.573036 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1095,"mutex_wait_us":2,"thread_start_us":298,"threads_started":1}
I20260928 08:42:14.575829 20118 tablet_service.cc:1467] Processing CreateTablet for tablet 04144f89ccf147269be6956b43df5c03 (DEFAULT_TABLE table=test-table [id=ed03e8aff11a4b7eb8944a6a726bb818]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:42:14.576647 19985 tablet_service.cc:1467] Processing CreateTablet for tablet 04144f89ccf147269be6956b43df5c03 (DEFAULT_TABLE table=test-table [id=ed03e8aff11a4b7eb8944a6a726bb818]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:42:14.576648 19851 tablet_service.cc:1467] Processing CreateTablet for tablet 04144f89ccf147269be6956b43df5c03 (DEFAULT_TABLE table=test-table [id=ed03e8aff11a4b7eb8944a6a726bb818]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:42:14.577414 20118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04144f89ccf147269be6956b43df5c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:14.577934 19851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04144f89ccf147269be6956b43df5c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:14.577955 19985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04144f89ccf147269be6956b43df5c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:14.588975 20201 tablet_bootstrap.cc:492] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22: Bootstrap starting.
I20260928 08:42:14.592304 20202 tablet_bootstrap.cc:492] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad: Bootstrap starting.
I20260928 08:42:14.592764 20201 tablet_bootstrap.cc:654] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:14.593127 20203 tablet_bootstrap.cc:492] T 04144f89ccf147269be6956b43df5c03 P 35b3015bc34e4062b00c4b2732b932b9: Bootstrap starting.
I20260928 08:42:14.595467 20201 log.cc:824] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:14.596606 20202 tablet_bootstrap.cc:654] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:14.596738 20203 tablet_bootstrap.cc:654] T 04144f89ccf147269be6956b43df5c03 P 35b3015bc34e4062b00c4b2732b932b9: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:14.597707 20202 log.cc:824] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:14.597924 20203 log.cc:824] T 04144f89ccf147269be6956b43df5c03 P 35b3015bc34e4062b00c4b2732b932b9: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:14.598999 20201 tablet_bootstrap.cc:492] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22: No bootstrap required, opened a new log
I20260928 08:42:14.599416 20201 ts_tablet_manager.cc:1397] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260928 08:42:14.609252 20202 tablet_bootstrap.cc:492] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad: No bootstrap required, opened a new log
I20260928 08:42:14.609252 20203 tablet_bootstrap.cc:492] T 04144f89ccf147269be6956b43df5c03 P 35b3015bc34e4062b00c4b2732b932b9: No bootstrap required, opened a new log
I20260928 08:42:14.609591 20203 ts_tablet_manager.cc:1397] T 04144f89ccf147269be6956b43df5c03 P 35b3015bc34e4062b00c4b2732b932b9: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20260928 08:42:14.609589 20202 ts_tablet_manager.cc:1397] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20260928 08:42:14.615249 20201 raft_consensus.cc:348] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3015bc34e4062b00c4b2732b932b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } } peers { permanent_uuid: "43c51f97676b418398b9947560bbab22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40833 } } peers { permanent_uuid: "8400ba9e115243acaedbac0396c2dbad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43337 } }
I20260928 08:42:14.616191 20201 raft_consensus.cc:374] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:14.616353 20201 raft_consensus.cc:729] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c51f97676b418398b9947560bbab22, State: Initialized, Role: FOLLOWER
I20260928 08:42:14.617244 20201 consensus_queue.cc:260] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3015bc34e4062b00c4b2732b932b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } } peers { permanent_uuid: "43c51f97676b418398b9947560bbab22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40833 } } peers { permanent_uuid: "8400ba9e115243acaedbac0396c2dbad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43337 } }
I20260928 08:42:14.618584 20183 heartbeater.cc:503] Master 127.0.97.62:40437 was elected leader, sending a full tablet report...
I20260928 08:42:14.619789 20201 ts_tablet_manager.cc:1428] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22: Time spent starting tablet: real 0.020s user 0.015s sys 0.000s
I20260928 08:42:14.621340 20202 raft_consensus.cc:348] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3015bc34e4062b00c4b2732b932b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } } peers { permanent_uuid: "43c51f97676b418398b9947560bbab22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40833 } } peers { permanent_uuid: "8400ba9e115243acaedbac0396c2dbad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43337 } }
I20260928 08:42:14.621336 20203 raft_consensus.cc:348] T 04144f89ccf147269be6956b43df5c03 P 35b3015bc34e4062b00c4b2732b932b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3015bc34e4062b00c4b2732b932b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } } peers { permanent_uuid: "43c51f97676b418398b9947560bbab22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40833 } } peers { permanent_uuid: "8400ba9e115243acaedbac0396c2dbad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43337 } }
I20260928 08:42:14.621848 20202 raft_consensus.cc:374] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:14.621835 20203 raft_consensus.cc:374] T 04144f89ccf147269be6956b43df5c03 P 35b3015bc34e4062b00c4b2732b932b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:14.621974 20202 raft_consensus.cc:729] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8400ba9e115243acaedbac0396c2dbad, State: Initialized, Role: FOLLOWER
I20260928 08:42:14.622088 20203 raft_consensus.cc:729] T 04144f89ccf147269be6956b43df5c03 P 35b3015bc34e4062b00c4b2732b932b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b3015bc34e4062b00c4b2732b932b9, State: Initialized, Role: FOLLOWER
I20260928 08:42:14.622859 20203 consensus_queue.cc:260] T 04144f89ccf147269be6956b43df5c03 P 35b3015bc34e4062b00c4b2732b932b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3015bc34e4062b00c4b2732b932b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } } peers { permanent_uuid: "43c51f97676b418398b9947560bbab22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40833 } } peers { permanent_uuid: "8400ba9e115243acaedbac0396c2dbad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43337 } }
I20260928 08:42:14.622855 20202 consensus_queue.cc:260] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3015bc34e4062b00c4b2732b932b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } } peers { permanent_uuid: "43c51f97676b418398b9947560bbab22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40833 } } peers { permanent_uuid: "8400ba9e115243acaedbac0396c2dbad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43337 } }
I20260928 08:42:14.623981 20050 heartbeater.cc:503] Master 127.0.97.62:40437 was elected leader, sending a full tablet report...
I20260928 08:42:14.625087 20202 ts_tablet_manager.cc:1428] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260928 08:42:14.626972 20203 ts_tablet_manager.cc:1428] T 04144f89ccf147269be6956b43df5c03 P 35b3015bc34e4062b00c4b2732b932b9: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260928 08:42:14.701246 20208 raft_consensus.cc:482] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:42:14.701619 20208 raft_consensus.cc:504] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3015bc34e4062b00c4b2732b932b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } } peers { permanent_uuid: "43c51f97676b418398b9947560bbab22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40833 } } peers { permanent_uuid: "8400ba9e115243acaedbac0396c2dbad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43337 } }
I20260928 08:42:14.703380 20208 leader_election.cc:288] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35b3015bc34e4062b00c4b2732b932b9 (127.0.97.1:45041), 43c51f97676b418398b9947560bbab22 (127.0.97.3:40833)
I20260928 08:42:14.712867 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:14.703813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38081 (local address 127.0.97.3:40833)
0928 08:42:14.704015 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:14.704023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:14.704796 (+ 773us) server_negotiation.cc:407] Connection header received
0928 08:42:14.704991 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:14.704997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:14.705081 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:14.705192 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:14.706087 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.707322 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.710357 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.711654 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.711786 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:14.711895 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:14.712462 (+ 567us) server_negotiation.cc:299] Negotiation successful
0928 08:42:14.712604 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:42:14.712870 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:14.703890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42615 (local address 127.0.97.1:45041)
0928 08:42:14.704084 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:14.704091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:14.704797 (+ 706us) server_negotiation.cc:407] Connection header received
0928 08:42:14.704991 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:14.704997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:14.705080 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:14.705196 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:14.706085 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.707332 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.710359 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.711652 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.711785 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:14.711895 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:14.712462 (+ 567us) server_negotiation.cc:299] Negotiation successful
0928 08:42:14.712604 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:42:14.713158 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:14.703705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40833 (local address 127.0.97.2:38081)
0928 08:42:14.704629 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:14.704656 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:14.704840 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:14.705222 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:14.705227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:14.705258 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:14.705946 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:14.705959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.707455 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.707461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:14.710187 (+ 2726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:14.710204 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.711928 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.711935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:14.712250 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:14.712317 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 08:42:14.712418 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 08:42:14.712638 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":350,"spinlock_wait_cycles":19456,"thread_start_us":493,"threads_started":1}
I20260928 08:42:14.713181 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:14.703781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.97.2:42615)
0928 08:42:14.704668 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:14.704686 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:14.704840 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:14.705222 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:14.705227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:14.705261 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:14.705855 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:14.705875 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.707457 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.707461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:14.710187 (+ 2726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:14.710204 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.711931 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.711935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:14.712226 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:14.712316 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 08:42:14.712418 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 08:42:14.712629 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":218,"spinlock_wait_cycles":33792,"thread_start_us":515,"threads_started":1}
I20260928 08:42:14.714257 20138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04144f89ccf147269be6956b43df5c03" candidate_uuid: "8400ba9e115243acaedbac0396c2dbad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c51f97676b418398b9947560bbab22" is_pre_election: true
I20260928 08:42:14.714315 19871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04144f89ccf147269be6956b43df5c03" candidate_uuid: "8400ba9e115243acaedbac0396c2dbad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b3015bc34e4062b00c4b2732b932b9" is_pre_election: true
I20260928 08:42:14.714882 19871 raft_consensus.cc:2454] T 04144f89ccf147269be6956b43df5c03 P 35b3015bc34e4062b00c4b2732b932b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8400ba9e115243acaedbac0396c2dbad in term 0.
I20260928 08:42:14.714884 20138 raft_consensus.cc:2454] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8400ba9e115243acaedbac0396c2dbad in term 0.
I20260928 08:42:14.715595 19939 leader_election.cc:302] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 35b3015bc34e4062b00c4b2732b932b9, 43c51f97676b418398b9947560bbab22, 8400ba9e115243acaedbac0396c2dbad; no voters:
I20260928 08:42:14.716049 20208 raft_consensus.cc:2781] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:42:14.716202 20208 raft_consensus.cc:482] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:42:14.716301 20208 raft_consensus.cc:3037] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:14.719081 20208 raft_consensus.cc:504] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3015bc34e4062b00c4b2732b932b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } } peers { permanent_uuid: "43c51f97676b418398b9947560bbab22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40833 } } peers { permanent_uuid: "8400ba9e115243acaedbac0396c2dbad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43337 } }
I20260928 08:42:14.719746 20208 leader_election.cc:288] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [CANDIDATE]: Term 1 election: Requested vote from peers 35b3015bc34e4062b00c4b2732b932b9 (127.0.97.1:45041), 43c51f97676b418398b9947560bbab22 (127.0.97.3:40833)
I20260928 08:42:14.720177 19871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04144f89ccf147269be6956b43df5c03" candidate_uuid: "8400ba9e115243acaedbac0396c2dbad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b3015bc34e4062b00c4b2732b932b9"
I20260928 08:42:14.720225 20138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04144f89ccf147269be6956b43df5c03" candidate_uuid: "8400ba9e115243acaedbac0396c2dbad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c51f97676b418398b9947560bbab22"
I20260928 08:42:14.720419 20138 raft_consensus.cc:3037] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:14.720418 19871 raft_consensus.cc:3037] T 04144f89ccf147269be6956b43df5c03 P 35b3015bc34e4062b00c4b2732b932b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:14.723757 19871 raft_consensus.cc:2454] T 04144f89ccf147269be6956b43df5c03 P 35b3015bc34e4062b00c4b2732b932b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8400ba9e115243acaedbac0396c2dbad in term 1.
I20260928 08:42:14.723757 20138 raft_consensus.cc:2454] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8400ba9e115243acaedbac0396c2dbad in term 1.
I20260928 08:42:14.724217 19939 leader_election.cc:302] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35b3015bc34e4062b00c4b2732b932b9, 8400ba9e115243acaedbac0396c2dbad; no voters:
I20260928 08:42:14.724529 20208 raft_consensus.cc:2781] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:14.725039 20208 raft_consensus.cc:686] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [term 1 LEADER]: Becoming Leader. State: Replica: 8400ba9e115243acaedbac0396c2dbad, State: Running, Role: LEADER
W20260928 08:42:14.725325 19917 tablet.cc:2369] T 04144f89ccf147269be6956b43df5c03 P 35b3015bc34e4062b00c4b2732b932b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:42:14.725625 20208 consensus_queue.cc:237] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3015bc34e4062b00c4b2732b932b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } } peers { permanent_uuid: "43c51f97676b418398b9947560bbab22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40833 } } peers { permanent_uuid: "8400ba9e115243acaedbac0396c2dbad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43337 } }
W20260928 08:42:14.730188 20051 tablet.cc:2369] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:42:14.730538 19727 catalog_manager.cc:5495] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad reported cstate change: term changed from 0 to 1, leader changed from <none> to 8400ba9e115243acaedbac0396c2dbad (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8400ba9e115243acaedbac0396c2dbad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b3015bc34e4062b00c4b2732b932b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43c51f97676b418398b9947560bbab22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8400ba9e115243acaedbac0396c2dbad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43337 } health_report { overall_health: HEALTHY } } }
W20260928 08:42:14.738418 20184 tablet.cc:2369] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:42:14.766251 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:14.761401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60550 (local address 127.0.97.2:43337)
0928 08:42:14.761582 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:14.761586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:14.761604 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:42:14.761714 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:14.761718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:14.761803 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:14.761924 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:42:14.762229 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.763004 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.764352 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.764753 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.764843 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:14.765273 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:42:14.765774 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:42:14.765880 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:14.765962 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 08:42:14.766026 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 08:42:14.774482 20138 raft_consensus.cc:1261] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22 [term 1 FOLLOWER]: Refusing update from remote peer 8400ba9e115243acaedbac0396c2dbad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:42:14.774482 19871 raft_consensus.cc:1261] T 04144f89ccf147269be6956b43df5c03 P 35b3015bc34e4062b00c4b2732b932b9 [term 1 FOLLOWER]: Refusing update from remote peer 8400ba9e115243acaedbac0396c2dbad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:42:14.775611 20208 consensus_queue.cc:1035] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [LEADER]: Connected to new peer: Peer: permanent_uuid: "43c51f97676b418398b9947560bbab22" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:42:14.775928 20212 consensus_queue.cc:1035] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b3015bc34e4062b00c4b2732b932b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:42:14.787561 20215 mvcc.cc:205] Tried to move back new op lower bound from 7334235892825550848 to 7334235892640841728. Current Snapshot: MvccSnapshot[applied={T|T < 7334235892825550848}]
I20260928 08:42:14.790443 20216 mvcc.cc:205] Tried to move back new op lower bound from 7334235892825550848 to 7334235892640841728. Current Snapshot: MvccSnapshot[applied={T|T < 7334235892825550848}]
I20260928 08:42:14.800439 19727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:42:14.800758 19727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:42:14.804251 19727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0646c471056c4db79e42830b8ce3f69c: Sending DeleteTablet for 3 replicas of tablet 04144f89ccf147269be6956b43df5c03
I20260928 08:42:14.805413 20118 tablet_service.cc:1514] Processing DeleteTablet for tablet 04144f89ccf147269be6956b43df5c03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:42:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43010
I20260928 08:42:14.805421 19985 tablet_service.cc:1514] Processing DeleteTablet for tablet 04144f89ccf147269be6956b43df5c03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:42:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60534
I20260928 08:42:14.805521 19851 tablet_service.cc:1514] Processing DeleteTablet for tablet 04144f89ccf147269be6956b43df5c03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:42:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39678
I20260928 08:42:14.808478 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 19788
I20260928 08:42:14.820078 20224 tablet_replica.cc:330] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad: stopping tablet replica
I20260928 08:42:14.820643 20224 raft_consensus.cc:2229] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:42:14.821390 20224 raft_consensus.cc:2258] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:42:14.822269 20225 tablet_replica.cc:330] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22: stopping tablet replica
I20260928 08:42:14.822888 20225 raft_consensus.cc:2229] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:42:14.823503 20225 raft_consensus.cc:2258] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:42:14.827401 20225 ts_tablet_manager.cc:1905] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:42:14.833869 20225 ts_tablet_manager.cc:1918] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:42:14.833999 20225 log.cc:1196] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/04144f89ccf147269be6956b43df5c03
I20260928 08:42:14.834446 20225 ts_tablet_manager.cc:1939] T 04144f89ccf147269be6956b43df5c03 P 43c51f97676b418398b9947560bbab22: Deleting consensus metadata
I20260928 08:42:14.836350 19711 catalog_manager.cc:4841] TS 43c51f97676b418398b9947560bbab22 (127.0.97.3:40833): tablet 04144f89ccf147269be6956b43df5c03 (table test-table [id=ed03e8aff11a4b7eb8944a6a726bb818]) successfully deleted
I20260928 08:42:14.836884 20224 ts_tablet_manager.cc:1905] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:42:14.844514 20224 ts_tablet_manager.cc:1918] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:42:14.844715 20224 log.cc:1196] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/04144f89ccf147269be6956b43df5c03
I20260928 08:42:14.845260 20224 ts_tablet_manager.cc:1939] T 04144f89ccf147269be6956b43df5c03 P 8400ba9e115243acaedbac0396c2dbad: Deleting consensus metadata
Sep 28 08:42:14 dist-test-slave-w6c0 krb5kdc[19682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:42:14.845991 20226 tablet_replica.cc:330] T 04144f89ccf147269be6956b43df5c03 P 35b3015bc34e4062b00c4b2732b932b9: stopping tablet replica
I20260928 08:42:14.846536 20226 raft_consensus.cc:2229] T 04144f89ccf147269be6956b43df5c03 P 35b3015bc34e4062b00c4b2732b932b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:42:14.847087 19712 catalog_manager.cc:4841] TS 8400ba9e115243acaedbac0396c2dbad (127.0.97.2:43337): tablet 04144f89ccf147269be6956b43df5c03 (table test-table [id=ed03e8aff11a4b7eb8944a6a726bb818]) successfully deleted
I20260928 08:42:14.847185 20226 raft_consensus.cc:2258] T 04144f89ccf147269be6956b43df5c03 P 35b3015bc34e4062b00c4b2732b932b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:42:14.851841 20226 ts_tablet_manager.cc:1905] T 04144f89ccf147269be6956b43df5c03 P 35b3015bc34e4062b00c4b2732b932b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:42:14.854238 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:14.810052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.0.97.1:45041)
0928 08:42:14.810247 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:14.810254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:14.825588 (+ 15334us) server_negotiation.cc:407] Connection header received
0928 08:42:14.825889 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:14.825895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:14.826050 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:14.826138 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:14.836922 (+ 10784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.838128 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.840148 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.840824 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.840925 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:14.847004 (+ 6079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:14.847022 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:14.847044 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:14.847092 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:14.850863 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:14.851735 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:14.851745 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:14.851749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:14.851925 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:14.852353 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:14.852360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:14.852364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:14.852928 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:14.853257 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:14.853552 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 08:42:14.853805 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:42:14.855017 19911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:14.969105 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 19788
I20260928 08:42:14.986855 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 19922
W20260928 08:42:14.986856 19712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45041 (error 108)
Sep 28 08:42:14 dist-test-slave-w6c0 krb5kdc[19682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:42:15.002899 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:15.002354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.0.1:39692)
0928 08:42:15.002564 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:15.002686 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 08:42:15.003396 19712 catalog_manager.cc:4584] TS 35b3015bc34e4062b00c4b2732b932b9 (127.0.97.1:45041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 04144f89ccf147269be6956b43df5c03: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45041: connect: Connection refused (error 111)
I20260928 08:42:15.009740 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:14.988418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.0.97.2:43337)
0928 08:42:14.988670 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:14.988679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:14.989046 (+ 367us) server_negotiation.cc:407] Connection header received
0928 08:42:14.989357 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:14.989370 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:14.989491 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:14.989590 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:14.991985 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.993689 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.996274 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:14.997805 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:14.997961 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:15.003211 (+ 5250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:15.003225 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:15.003242 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:15.003287 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:15.006600 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:15.007454 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:15.007463 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:15.007467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:15.007620 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:15.008098 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:15.008104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:15.008108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:15.008559 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:15.008938 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:15.009133 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 08:42:15.009357 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:42:15.010396 20045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:15.057228 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:15.056753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.0.1:39700)
0928 08:42:15.056973 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:15.057072 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 08:42:15.058192 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:15.057851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.0.1:39714)
0928 08:42:15.058008 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:15.058077 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 08:42:15.118594 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:15.118083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.0.1:39720)
0928 08:42:15.118286 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:15.118381 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 08:42:15.119557 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:15.119215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.0.1:39732)
0928 08:42:15.119357 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:15.119435 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 08:42:15.124373 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 19922
I20260928 08:42:15.143720 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 20055
Sep 28 08:42:15 dist-test-slave-w6c0 krb5kdc[19682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:42:15.158366 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:15.144429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43014 (local address 127.0.97.3:40833)
0928 08:42:15.144623 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:15.144629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:15.144650 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:42:15.144840 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:15.144846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:15.144922 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:15.145039 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:15.146319 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:15.147119 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:15.148336 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:15.148842 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:15.148970 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:15.153289 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:15.153302 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:15.153319 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:15.153357 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:15.156129 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:15.156684 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:15.156690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:15.156693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:15.156819 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:15.157137 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:15.157140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:15.157142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:15.157468 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:15.157679 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:15.157947 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 08:42:15.158098 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:42:15.159067 20178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:15.201110 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:15.200367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.0.1:39742)
0928 08:42:15.200724 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:15.200860 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260928 08:42:15.202239 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:15.201816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.0.1:39758)
0928 08:42:15.202001 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:15.202110 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 08:42:15.253315 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 20055
I20260928 08:42:15.269963 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 19695
I20260928 08:42:15.271119 19757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:15.381712 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:15.381177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.0.1:39772)
0928 08:42:15.381378 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:15.381488 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 08:42:15.382031 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 19695
2026-09-28T08:42:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4299 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T08:42:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:15Z Disabled control of system clock
I20260928 08:42:15.458848 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44295
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:42:15.781133 20236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:15.781419 20236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:15.781500 20236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:15.788688 20236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:15.788794 20236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:15.788833 20236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:15.788892 20236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:15.788936 20236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:15.798060 20236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44295
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:15.799404 20236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:15.801172 20236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:15.812912 20244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:15.812975 20242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:15.813496 20241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:15.813766 20236 server_base.cc:1034] running on GCE node
I20260928 08:42:15.814697 20236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:15.816087 20236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:15.817312 20236 hybrid_clock.cc:648] HybridClock initialized: now 1790584935817271 us; error 38 us; skew 500 ppm
I20260928 08:42:15.817672 20236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:42:15.825481 20236 webserver.cc:460] Webserver started at http://127.0.97.62:34597/ using document root <none> and password file <none>
I20260928 08:42:15.826186 20236 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:15.826313 20236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:15.826697 20236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:15.829548 20236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "b636956a86d74953b32a61f427329687"
format_stamp: "Formatted at 2026-09-28 08:42:15 on dist-test-slave-w6c0"
server_key: "cf2c983e878f6f99606fd38c22b23087"
server_key_iv: "4d92b64e5831b299a40190a03116a022"
server_key_version: "encryptionkey@0"
I20260928 08:42:15.830336 20236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "b636956a86d74953b32a61f427329687"
format_stamp: "Formatted at 2026-09-28 08:42:15 on dist-test-slave-w6c0"
server_key: "cf2c983e878f6f99606fd38c22b23087"
server_key_iv: "4d92b64e5831b299a40190a03116a022"
server_key_version: "encryptionkey@0"
I20260928 08:42:15.835315 20236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 08:42:15.838660 20250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:15.839936 20236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:42:15.840119 20236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "b636956a86d74953b32a61f427329687"
format_stamp: "Formatted at 2026-09-28 08:42:15 on dist-test-slave-w6c0"
server_key: "cf2c983e878f6f99606fd38c22b23087"
server_key_iv: "4d92b64e5831b299a40190a03116a022"
server_key_version: "encryptionkey@0"
I20260928 08:42:15.840328 20236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:15.856987 20236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:15.857815 20236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:15.858099 20236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:15.876731 20236 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44295
I20260928 08:42:15.876757 20301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44295 every 8 connection(s)
I20260928 08:42:15.878032 20236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:15.880035 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 20236
I20260928 08:42:15.880218 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:15.880877 388 external_mini_cluster.cc:1428] Setting key e506b214ada545b34a45f9a608981aad
I20260928 08:42:15.882478 20302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:15.893723 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:15.882901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.0.97.62:44295)
0928 08:42:15.883855 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:15.883865 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:15.883904 (+ 39us) server_negotiation.cc:407] Connection header received
0928 08:42:15.885467 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:15.885487 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:15.885909 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:15.886300 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:15.886859 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:15.888286 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:15.889634 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:15.890258 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:15.890463 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:15.891600 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:15.891607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:15.891620 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:15.892164 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:15.892237 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:15.892260 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:15.892395 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:15.892616 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 08:42:15.892886 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":378,"threads_started":1}
I20260928 08:42:15.894577 20302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687: Bootstrap starting.
I20260928 08:42:15.897709 20302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:15.898691 20302 log.cc:824] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:15.901162 20302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687: No bootstrap required, opened a new log
I20260928 08:42:15.908450 20302 raft_consensus.cc:348] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b636956a86d74953b32a61f427329687" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44295 } }
I20260928 08:42:15.908761 20302 raft_consensus.cc:374] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:15.908838 20302 raft_consensus.cc:729] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b636956a86d74953b32a61f427329687, State: Initialized, Role: FOLLOWER
I20260928 08:42:15.909423 20302 consensus_queue.cc:260] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b636956a86d74953b32a61f427329687" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44295 } }
I20260928 08:42:15.909664 20302 raft_consensus.cc:388] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:15.909811 20302 raft_consensus.cc:482] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:15.909978 20302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:15.912079 20302 raft_consensus.cc:504] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b636956a86d74953b32a61f427329687" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44295 } }
I20260928 08:42:15.912561 20302 leader_election.cc:302] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b636956a86d74953b32a61f427329687; no voters:
I20260928 08:42:15.913146 20302 leader_election.cc:288] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:15.913277 20307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:15.914022 20307 raft_consensus.cc:686] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687 [term 1 LEADER]: Becoming Leader. State: Replica: b636956a86d74953b32a61f427329687, State: Running, Role: LEADER
I20260928 08:42:15.914636 20307 consensus_queue.cc:237] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b636956a86d74953b32a61f427329687" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44295 } }
I20260928 08:42:15.915205 20302 sys_catalog.cc:564] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:15.917182 20309 sys_catalog.cc:455] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b636956a86d74953b32a61f427329687. Latest consensus state: current_term: 1 leader_uuid: "b636956a86d74953b32a61f427329687" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b636956a86d74953b32a61f427329687" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44295 } } }
I20260928 08:42:15.917590 20309 sys_catalog.cc:458] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:15.918215 20308 sys_catalog.cc:455] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b636956a86d74953b32a61f427329687" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b636956a86d74953b32a61f427329687" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44295 } } }
I20260928 08:42:15.918677 20308 sys_catalog.cc:458] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:15.920784 20315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:15.925820 20315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:15.935398 20315 catalog_manager.cc:1347] Generated new cluster ID: e72033aed9c848a780597af36276b9c6
I20260928 08:42:15.935564 20315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:15.962343 20315 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:15.963671 20315 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:15.988256 20315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687: Generated new TSK 0
I20260928 08:42:15.989142 20315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:16.007184 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44295
--builtin_ntp_servers=127.0.97.20:35587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:42:16.356726 20326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:16.357025 20326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:16.357185 20326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:16.364517 20326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:16.364647 20326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:16.364823 20326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:42:16.374526 20326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:16.375860 20326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:16.377578 20326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:16.390763 20331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:16.390681 20332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:16.390725 20334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:16.391585 20326 server_base.cc:1034] running on GCE node
I20260928 08:42:16.392143 20326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:16.393105 20326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:16.394335 20326 hybrid_clock.cc:648] HybridClock initialized: now 1790584936394295 us; error 35 us; skew 500 ppm
I20260928 08:42:16.394716 20326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:42:16.396946 20326 webserver.cc:460] Webserver started at http://127.0.97.1:33995/ using document root <none> and password file <none>
I20260928 08:42:16.397636 20326 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:16.397759 20326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:16.398108 20326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:16.400983 20326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-0/data/instance:
uuid: "3a32a73c9a2a4c0583131315ae1f70f8"
format_stamp: "Formatted at 2026-09-28 08:42:16 on dist-test-slave-w6c0"
server_key: "0a75dc7a82cd3329f105a946fc230f7b"
server_key_iv: "3d5e9054f89bd05861d8528bff206fac"
server_key_version: "encryptionkey@0"
I20260928 08:42:16.401857 20326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3a32a73c9a2a4c0583131315ae1f70f8"
format_stamp: "Formatted at 2026-09-28 08:42:16 on dist-test-slave-w6c0"
server_key: "0a75dc7a82cd3329f105a946fc230f7b"
server_key_iv: "3d5e9054f89bd05861d8528bff206fac"
server_key_version: "encryptionkey@0"
I20260928 08:42:16.406944 20326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 08:42:16.410137 20340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:16.411378 20326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:42:16.411543 20326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-0/wal
uuid: "3a32a73c9a2a4c0583131315ae1f70f8"
format_stamp: "Formatted at 2026-09-28 08:42:16 on dist-test-slave-w6c0"
server_key: "0a75dc7a82cd3329f105a946fc230f7b"
server_key_iv: "3d5e9054f89bd05861d8528bff206fac"
server_key_version: "encryptionkey@0"
I20260928 08:42:16.411746 20326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:16.458115 20326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:16.458998 20326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:16.459285 20326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:16.460311 20326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:42:16.461784 20326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:42:16.461907 20326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:16.462038 20326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:42:16.462127 20326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:16.494091 20326 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46811
I20260928 08:42:16.494133 20452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46811 every 8 connection(s)
I20260928 08:42:16.495393 20326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:42:16.496368 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 20326
I20260928 08:42:16.496546 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:42:16.496978 388 external_mini_cluster.cc:1428] Setting key 205ff650a8e71903db2f836cd6092551
I20260928 08:42:16.499914 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44295
--builtin_ntp_servers=127.0.97.20:35587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 08:42:16.508260 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:16.498084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33861 (local address 127.0.97.62:44295)
0928 08:42:16.498580 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:16.498586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:16.501765 (+ 3179us) server_negotiation.cc:407] Connection header received
0928 08:42:16.501821 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:16.501824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:16.501886 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:16.502003 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:16.503113 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:16.503848 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:16.505353 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:16.505751 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:16.505877 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:16.507252 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:16.507261 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:16.507269 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:16.507295 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:16.507349 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:16.507368 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:16.507458 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:16.507709 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 08:42:16.507819 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":233,"threads_started":1}
I20260928 08:42:16.508984 20456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:16.498778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44295 (local address 127.0.97.1:33861)
0928 08:42:16.499809 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:16.499854 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:16.501501 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:16.502258 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:16.502265 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:16.502349 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:16.502951 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:16.502968 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:16.504000 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:16.504006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:16.505206 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:16.505218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:16.505977 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:16.505984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:16.506213 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:16.506918 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:42:16.506921 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:16.506957 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:42:16.507028 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:16.507479 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:16.507488 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:16.507495 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 08:42:16.507654 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 08:42:16.507929 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":348,"threads_started":1}
I20260928 08:42:16.510533 20453 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44295
I20260928 08:42:16.510838 20453 heartbeater.cc:464] Registering TS with master...
I20260928 08:42:16.511669 20453 heartbeater.cc:511] Master 127.0.97.62:44295 requested a full tablet report, sending...
I20260928 08:42:16.513695 20267 ts_manager.cc:194] Registered new tserver with Master: 3a32a73c9a2a4c0583131315ae1f70f8 (127.0.97.1:46811)
I20260928 08:42:16.515319 20267 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:33861
W20260928 08:42:16.821014 20457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:16.821344 20457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:16.821501 20457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:16.828845 20457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:16.828969 20457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:16.829145 20457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:42:16.838235 20457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:16.839564 20457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:16.841270 20457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:16.854445 20462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:16.854408 20465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:16.854480 20463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:16.855275 20457 server_base.cc:1034] running on GCE node
I20260928 08:42:16.855847 20457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:16.856758 20457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:16.857985 20457 hybrid_clock.cc:648] HybridClock initialized: now 1790584936857927 us; error 60 us; skew 500 ppm
I20260928 08:42:16.858296 20457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:42:16.860473 20457 webserver.cc:460] Webserver started at http://127.0.97.2:43567/ using document root <none> and password file <none>
I20260928 08:42:16.861140 20457 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:16.861255 20457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:16.861579 20457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:16.864547 20457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-1/data/instance:
uuid: "fe82770a836c4f5ab0c9503b8014b122"
format_stamp: "Formatted at 2026-09-28 08:42:16 on dist-test-slave-w6c0"
server_key: "45a583077955541b1fba355482392054"
server_key_iv: "b29511c6a55319e665e80035f21f08cf"
server_key_version: "encryptionkey@0"
I20260928 08:42:16.865324 20457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fe82770a836c4f5ab0c9503b8014b122"
format_stamp: "Formatted at 2026-09-28 08:42:16 on dist-test-slave-w6c0"
server_key: "45a583077955541b1fba355482392054"
server_key_iv: "b29511c6a55319e665e80035f21f08cf"
server_key_version: "encryptionkey@0"
I20260928 08:42:16.870229 20457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 08:42:16.873451 20471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:16.874728 20457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:42:16.874884 20457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-1/wal
uuid: "fe82770a836c4f5ab0c9503b8014b122"
format_stamp: "Formatted at 2026-09-28 08:42:16 on dist-test-slave-w6c0"
server_key: "45a583077955541b1fba355482392054"
server_key_iv: "b29511c6a55319e665e80035f21f08cf"
server_key_version: "encryptionkey@0"
I20260928 08:42:16.875098 20457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:16.898983 20457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:16.899778 20457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:16.900097 20457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:16.901016 20457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:42:16.902417 20457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:42:16.902563 20457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:16.902705 20457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:42:16.902792 20457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:16.935410 20457 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34123
I20260928 08:42:16.935452 20583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34123 every 8 connection(s)
I20260928 08:42:16.936679 20457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:42:16.945719 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 20457
I20260928 08:42:16.945910 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:42:16.946437 388 external_mini_cluster.cc:1428] Setting key 6f8fa92d537f7e3135901f7ea8130a7e
I20260928 08:42:16.948925 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:16.939208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40515 (local address 127.0.97.62:44295)
0928 08:42:16.939440 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:16.939446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:16.940700 (+ 1254us) server_negotiation.cc:407] Connection header received
0928 08:42:16.942389 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:16.942394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:16.942460 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:16.942597 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:16.943550 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:16.944293 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:16.945795 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:16.946406 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:16.946568 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:16.948110 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:16.948115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:16.948118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:16.948135 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:16.948166 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:16.948178 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:16.948282 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:16.948525 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 08:42:16.948632 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 08:42:16.949532 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44295
--builtin_ntp_servers=127.0.97.20:35587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 08:42:16.949949 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:16.939660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44295 (local address 127.0.97.2:40515)
0928 08:42:16.940533 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:16.940594 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:16.942166 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:16.942755 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:16.942763 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:16.942849 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:16.943416 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:16.943432 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:16.944440 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:16.944443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:16.945660 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:16.945671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:16.946702 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:16.946708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:16.946902 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:16.947719 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:42:16.947722 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:16.947774 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:42:16.947870 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:16.948289 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:16.948305 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:16.948311 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 08:42:16.948516 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 08:42:16.948832 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":276,"threads_started":1}
I20260928 08:42:16.951319 20584 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44295
I20260928 08:42:16.951624 20584 heartbeater.cc:464] Registering TS with master...
I20260928 08:42:16.952420 20584 heartbeater.cc:511] Master 127.0.97.62:44295 requested a full tablet report, sending...
I20260928 08:42:16.953864 20267 ts_manager.cc:194] Registered new tserver with Master: fe82770a836c4f5ab0c9503b8014b122 (127.0.97.2:34123)
I20260928 08:42:16.954767 20267 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:40515
W20260928 08:42:17.276713 20587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:17.277030 20587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:17.277182 20587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:17.284559 20587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:17.284696 20587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:17.284878 20587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:42:17.294032 20587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:17.295343 20587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:17.297133 20587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:17.310137 20592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:17.310173 20593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:17.310321 20595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:17.310842 20587 server_base.cc:1034] running on GCE node
I20260928 08:42:17.311468 20587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:17.312383 20587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:17.313609 20587 hybrid_clock.cc:648] HybridClock initialized: now 1790584937313572 us; error 56 us; skew 500 ppm
I20260928 08:42:17.313925 20587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:42:17.316556 20587 webserver.cc:460] Webserver started at http://127.0.97.3:35195/ using document root <none> and password file <none>
I20260928 08:42:17.317222 20587 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:17.317348 20587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:17.317664 20587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:17.320595 20587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-2/data/instance:
uuid: "f36294b184614b46ba8ad334bfe6e7a5"
format_stamp: "Formatted at 2026-09-28 08:42:17 on dist-test-slave-w6c0"
server_key: "482ea096c25804a239d67e90814d854b"
server_key_iv: "89a6ac8c4c1af9f23eafd17b909d2e2e"
server_key_version: "encryptionkey@0"
I20260928 08:42:17.321386 20587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f36294b184614b46ba8ad334bfe6e7a5"
format_stamp: "Formatted at 2026-09-28 08:42:17 on dist-test-slave-w6c0"
server_key: "482ea096c25804a239d67e90814d854b"
server_key_iv: "89a6ac8c4c1af9f23eafd17b909d2e2e"
server_key_version: "encryptionkey@0"
I20260928 08:42:17.326463 20587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:42:17.329716 20601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:17.330979 20587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:42:17.331146 20587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-2/wal
uuid: "f36294b184614b46ba8ad334bfe6e7a5"
format_stamp: "Formatted at 2026-09-28 08:42:17 on dist-test-slave-w6c0"
server_key: "482ea096c25804a239d67e90814d854b"
server_key_iv: "89a6ac8c4c1af9f23eafd17b909d2e2e"
server_key_version: "encryptionkey@0"
I20260928 08:42:17.331348 20587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:17.350797 20587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:17.351876 20587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:17.352142 20587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:17.353048 20587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:42:17.354700 20587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:42:17.354786 20587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:17.354861 20587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:42:17.354914 20587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:17.387476 20587 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39825
I20260928 08:42:17.387518 20713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39825 every 8 connection(s)
I20260928 08:42:17.388741 20587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:42:17.394665 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 20587
I20260928 08:42:17.394840 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:42:17.395263 388 external_mini_cluster.cc:1428] Setting key 62048abce8722e8813fc54baab67af61
I20260928 08:42:17.401221 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:17.391253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33065 (local address 127.0.97.62:44295)
0928 08:42:17.391434 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:17.391439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:17.392701 (+ 1262us) server_negotiation.cc:407] Connection header received
0928 08:42:17.394386 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:17.394394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:17.394522 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:17.394640 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:17.395764 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.396807 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.398807 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.399191 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.399292 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:17.400497 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:17.400502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:17.400505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:17.400522 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:17.400555 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:17.400566 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:17.400637 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:17.400904 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 08:42:17.400988 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:42:17.402386 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:17.391716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44295 (local address 127.0.97.3:33065)
0928 08:42:17.392557 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:17.392601 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:17.394142 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:17.394859 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:17.394867 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:17.394983 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:17.395577 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:17.395594 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.396967 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.396971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:17.398688 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:17.398701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.399367 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.399371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:17.399568 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:17.400208 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:42:17.400211 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:17.400247 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:42:17.400327 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:17.400678 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:17.400687 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:17.400694 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 08:42:17.400856 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 08:42:17.401177 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":262,"threads_started":1}
I20260928 08:42:17.403755 20714 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44295
I20260928 08:42:17.404013 20714 heartbeater.cc:464] Registering TS with master...
I20260928 08:42:17.404649 20714 heartbeater.cc:511] Master 127.0.97.62:44295 requested a full tablet report, sending...
I20260928 08:42:17.405898 20267 ts_manager.cc:194] Registered new tserver with Master: f36294b184614b46ba8ad334bfe6e7a5 (127.0.97.3:39825)
I20260928 08:42:17.406752 20267 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:33065
I20260928 08:42:17.410034 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:42:17.419524 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:17.413909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55876 (local address 127.0.97.62:44295)
0928 08:42:17.414113 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:17.414122 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:17.414476 (+ 354us) server_negotiation.cc:407] Connection header received
0928 08:42:17.414725 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:17.414731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:17.414827 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:17.414903 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:17.415436 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.416168 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.417477 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.417885 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.417980 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:17.418990 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:17.418995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:17.418998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:17.419013 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:17.419044 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:17.419055 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:17.419126 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:17.419234 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 08:42:17.419303 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:42:17.423298 20267 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:42:17.430125 20267 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:42:17.463758 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:17.441740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46811 (local address 127.0.0.1:52874)
0928 08:42:17.453840 (+ 12100us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:17.453901 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:17.454158 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:17.455486 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:17.455497 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:17.455539 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:17.456020 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:17.456037 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.458195 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.458203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:17.460730 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:17.460744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.462654 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.462660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:17.462944 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:17.463036 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 08:42:17.463167 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 08:42:17.463368 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11747,"mutex_wait_us":55,"thread_start_us":16018,"threads_started":1}
I20260928 08:42:17.464560 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:17.441952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.0.97.1:46811)
0928 08:42:17.452887 (+ 10935us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:17.452900 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:17.454402 (+ 1502us) server_negotiation.cc:407] Connection header received
0928 08:42:17.454561 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:17.454572 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:17.454882 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:17.455122 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:17.456282 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.458036 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.460952 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.462430 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.462598 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:17.462855 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:17.464030 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0928 08:42:17.464218 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10692,"thread_start_us":10929,"threads_started":1}
I20260928 08:42:17.466845 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:17.442183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.0.97.2:34123)
0928 08:42:17.457428 (+ 15245us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:17.457437 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:17.458291 (+ 854us) server_negotiation.cc:407] Connection header received
0928 08:42:17.458391 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:17.458399 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:17.458668 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:17.458853 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:17.459698 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.461676 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.464199 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.465698 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.465831 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:17.466050 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:17.466333 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 08:42:17.466531 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15091,"thread_start_us":15262,"threads_started":1}
I20260928 08:42:17.467299 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:17.441933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34123 (local address 127.0.0.1:40200)
0928 08:42:17.458017 (+ 16084us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:17.458036 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:17.458160 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:17.459059 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:17.459063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:17.459087 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:17.459492 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:17.459505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.461857 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.461867 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:17.463987 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:17.464002 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.465946 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.465953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:17.466148 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:17.466182 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 08:42:17.466275 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 08:42:17.466401 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15959,"mutex_wait_us":1,"thread_start_us":11381,"threads_started":1}
I20260928 08:42:17.468827 20388 tablet_service.cc:1467] Processing CreateTablet for tablet d11f738bdf634ea9965ccac381c17d9f (DEFAULT_TABLE table=test-table [id=b973376a2d474d0897ccaa91317dacf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:42:17.469328 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:17.458398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39825 (local address 127.0.0.1:34848)
0928 08:42:17.461879 (+ 3481us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:17.461896 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:17.462045 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:17.462885 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:17.462889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:17.462921 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:17.463366 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:17.463382 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.465262 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.465267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:17.467182 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:17.467199 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.468615 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.468621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:17.468780 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:17.468822 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 08:42:17.468902 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 08:42:17.469031 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3339,"thread_start_us":3213,"threads_started":1}
I20260928 08:42:17.470222 20388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d11f738bdf634ea9965ccac381c17d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:17.470269 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:17.454781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34848 (local address 127.0.97.3:39825)
0928 08:42:17.461569 (+ 6788us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:17.461576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:17.462010 (+ 434us) server_negotiation.cc:407] Connection header received
0928 08:42:17.462206 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:17.462213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:17.462419 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:17.462619 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:17.463619 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.465117 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.467410 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.468428 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.468582 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:17.468840 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:17.469701 (+ 861us) server_negotiation.cc:299] Negotiation successful
0928 08:42:17.469878 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6603,"thread_start_us":6447,"threads_started":1}
I20260928 08:42:17.471093 20519 tablet_service.cc:1467] Processing CreateTablet for tablet d11f738bdf634ea9965ccac381c17d9f (DEFAULT_TABLE table=test-table [id=b973376a2d474d0897ccaa91317dacf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:42:17.472241 20519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d11f738bdf634ea9965ccac381c17d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:17.473658 20649 tablet_service.cc:1467] Processing CreateTablet for tablet d11f738bdf634ea9965ccac381c17d9f (DEFAULT_TABLE table=test-table [id=b973376a2d474d0897ccaa91317dacf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:42:17.474995 20649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d11f738bdf634ea9965ccac381c17d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:17.483086 20732 tablet_bootstrap.cc:492] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122: Bootstrap starting.
I20260928 08:42:17.486361 20732 tablet_bootstrap.cc:654] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:17.487330 20732 log.cc:824] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:17.488250 20733 tablet_bootstrap.cc:492] T d11f738bdf634ea9965ccac381c17d9f P 3a32a73c9a2a4c0583131315ae1f70f8: Bootstrap starting.
I20260928 08:42:17.491694 20732 tablet_bootstrap.cc:492] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122: No bootstrap required, opened a new log
I20260928 08:42:17.491719 20733 tablet_bootstrap.cc:654] T d11f738bdf634ea9965ccac381c17d9f P 3a32a73c9a2a4c0583131315ae1f70f8: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:17.491961 20732 ts_tablet_manager.cc:1397] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260928 08:42:17.493585 20735 tablet_bootstrap.cc:492] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5: Bootstrap starting.
I20260928 08:42:17.494626 20733 log.cc:824] T d11f738bdf634ea9965ccac381c17d9f P 3a32a73c9a2a4c0583131315ae1f70f8: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:17.497718 20735 tablet_bootstrap.cc:654] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:17.499207 20735 log.cc:824] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:17.502418 20732 raft_consensus.cc:348] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a32a73c9a2a4c0583131315ae1f70f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46811 } } peers { permanent_uuid: "fe82770a836c4f5ab0c9503b8014b122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34123 } } peers { permanent_uuid: "f36294b184614b46ba8ad334bfe6e7a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } }
I20260928 08:42:17.502887 20732 raft_consensus.cc:374] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:17.502986 20732 raft_consensus.cc:729] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe82770a836c4f5ab0c9503b8014b122, State: Initialized, Role: FOLLOWER
I20260928 08:42:17.503705 20732 consensus_queue.cc:260] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a32a73c9a2a4c0583131315ae1f70f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46811 } } peers { permanent_uuid: "fe82770a836c4f5ab0c9503b8014b122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34123 } } peers { permanent_uuid: "f36294b184614b46ba8ad334bfe6e7a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } }
I20260928 08:42:17.505831 20733 tablet_bootstrap.cc:492] T d11f738bdf634ea9965ccac381c17d9f P 3a32a73c9a2a4c0583131315ae1f70f8: No bootstrap required, opened a new log
I20260928 08:42:17.506192 20733 ts_tablet_manager.cc:1397] T d11f738bdf634ea9965ccac381c17d9f P 3a32a73c9a2a4c0583131315ae1f70f8: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20260928 08:42:17.513922 20584 heartbeater.cc:503] Master 127.0.97.62:44295 was elected leader, sending a full tablet report...
I20260928 08:42:17.514649 20732 ts_tablet_manager.cc:1428] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122: Time spent starting tablet: real 0.023s user 0.012s sys 0.011s
I20260928 08:42:17.515450 20735 tablet_bootstrap.cc:492] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5: No bootstrap required, opened a new log
I20260928 08:42:17.515901 20735 ts_tablet_manager.cc:1397] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.004s
I20260928 08:42:17.517671 20733 raft_consensus.cc:348] T d11f738bdf634ea9965ccac381c17d9f P 3a32a73c9a2a4c0583131315ae1f70f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a32a73c9a2a4c0583131315ae1f70f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46811 } } peers { permanent_uuid: "fe82770a836c4f5ab0c9503b8014b122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34123 } } peers { permanent_uuid: "f36294b184614b46ba8ad334bfe6e7a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } }
I20260928 08:42:17.518138 20733 raft_consensus.cc:374] T d11f738bdf634ea9965ccac381c17d9f P 3a32a73c9a2a4c0583131315ae1f70f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:17.518253 20733 raft_consensus.cc:729] T d11f738bdf634ea9965ccac381c17d9f P 3a32a73c9a2a4c0583131315ae1f70f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a32a73c9a2a4c0583131315ae1f70f8, State: Initialized, Role: FOLLOWER
I20260928 08:42:17.518879 20453 heartbeater.cc:503] Master 127.0.97.62:44295 was elected leader, sending a full tablet report...
I20260928 08:42:17.519624 20733 consensus_queue.cc:260] T d11f738bdf634ea9965ccac381c17d9f P 3a32a73c9a2a4c0583131315ae1f70f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a32a73c9a2a4c0583131315ae1f70f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46811 } } peers { permanent_uuid: "fe82770a836c4f5ab0c9503b8014b122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34123 } } peers { permanent_uuid: "f36294b184614b46ba8ad334bfe6e7a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } }
I20260928 08:42:17.522261 20733 ts_tablet_manager.cc:1428] T d11f738bdf634ea9965ccac381c17d9f P 3a32a73c9a2a4c0583131315ae1f70f8: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260928 08:42:17.526196 20735 raft_consensus.cc:348] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a32a73c9a2a4c0583131315ae1f70f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46811 } } peers { permanent_uuid: "fe82770a836c4f5ab0c9503b8014b122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34123 } } peers { permanent_uuid: "f36294b184614b46ba8ad334bfe6e7a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } }
I20260928 08:42:17.526569 20735 raft_consensus.cc:374] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:17.526681 20735 raft_consensus.cc:729] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36294b184614b46ba8ad334bfe6e7a5, State: Initialized, Role: FOLLOWER
I20260928 08:42:17.527197 20735 consensus_queue.cc:260] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a32a73c9a2a4c0583131315ae1f70f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46811 } } peers { permanent_uuid: "fe82770a836c4f5ab0c9503b8014b122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34123 } } peers { permanent_uuid: "f36294b184614b46ba8ad334bfe6e7a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } }
I20260928 08:42:17.528122 20714 heartbeater.cc:503] Master 127.0.97.62:44295 was elected leader, sending a full tablet report...
I20260928 08:42:17.528549 20735 ts_tablet_manager.cc:1428] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20260928 08:42:17.640453 20715 tablet.cc:2369] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:42:17.688760 20585 tablet.cc:2369] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:42:17.693693 20740 raft_consensus.cc:482] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:42:17.693982 20740 raft_consensus.cc:504] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a32a73c9a2a4c0583131315ae1f70f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46811 } } peers { permanent_uuid: "fe82770a836c4f5ab0c9503b8014b122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34123 } } peers { permanent_uuid: "f36294b184614b46ba8ad334bfe6e7a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } }
I20260928 08:42:17.695756 20740 leader_election.cc:288] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a32a73c9a2a4c0583131315ae1f70f8 (127.0.97.1:46811), fe82770a836c4f5ab0c9503b8014b122 (127.0.97.2:34123)
I20260928 08:42:17.702193 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:17.696314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44433 (local address 127.0.97.2:34123)
0928 08:42:17.696502 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:17.696508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:17.696532 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:42:17.696622 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:17.696628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:17.696735 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:17.696863 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:17.697447 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.698444 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.700306 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.701358 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.701453 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:17.701534 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:17.701904 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 08:42:17.702006 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 08:42:17.702314 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:17.696024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34123 (local address 127.0.97.3:44433)
0928 08:42:17.696185 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:17.696206 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:17.696337 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:17.697008 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:17.697013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:17.697043 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:17.697256 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:17.697271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.698612 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.698617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:17.700148 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:17.700159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.701506 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.701512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:17.701743 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:17.701782 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 08:42:17.701880 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 08:42:17.702027 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":3}
I20260928 08:42:17.703227 20741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:17.696024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46811 (local address 127.0.97.3:54181)
0928 08:42:17.697022 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:17.697041 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:17.697186 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:17.697628 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:17.697632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:17.697653 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:42:17.698033 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:17.698045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.699265 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.699269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:17.700903 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:17.700917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.702550 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.702559 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:17.702750 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:17.702782 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 08:42:17.702884 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 08:42:17.702999 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":147,"thread_start_us":481,"threads_started":1}
I20260928 08:42:17.703341 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:17.696301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54181 (local address 127.0.97.1:46811)
0928 08:42:17.696513 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:17.696519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:17.697136 (+ 617us) server_negotiation.cc:407] Connection header received
0928 08:42:17.697426 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:17.697433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:17.697511 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:17.697615 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:42:17.698186 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.699115 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.701091 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.702174 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.702293 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:17.702394 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:17.702936 (+ 542us) server_negotiation.cc:299] Negotiation successful
0928 08:42:17.703092 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:42:17.703481 20539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d11f738bdf634ea9965ccac381c17d9f" candidate_uuid: "f36294b184614b46ba8ad334bfe6e7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe82770a836c4f5ab0c9503b8014b122" is_pre_election: true
I20260928 08:42:17.704018 20539 raft_consensus.cc:2454] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f36294b184614b46ba8ad334bfe6e7a5 in term 0.
I20260928 08:42:17.704236 20408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d11f738bdf634ea9965ccac381c17d9f" candidate_uuid: "f36294b184614b46ba8ad334bfe6e7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a32a73c9a2a4c0583131315ae1f70f8" is_pre_election: true
I20260928 08:42:17.704674 20605 leader_election.cc:302] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f36294b184614b46ba8ad334bfe6e7a5, fe82770a836c4f5ab0c9503b8014b122; no voters:
I20260928 08:42:17.704773 20408 raft_consensus.cc:2454] T d11f738bdf634ea9965ccac381c17d9f P 3a32a73c9a2a4c0583131315ae1f70f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f36294b184614b46ba8ad334bfe6e7a5 in term 0.
I20260928 08:42:17.705116 20740 raft_consensus.cc:2781] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:42:17.705317 20740 raft_consensus.cc:482] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:42:17.705438 20740 raft_consensus.cc:3037] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:17.707703 20740 raft_consensus.cc:504] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a32a73c9a2a4c0583131315ae1f70f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46811 } } peers { permanent_uuid: "fe82770a836c4f5ab0c9503b8014b122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34123 } } peers { permanent_uuid: "f36294b184614b46ba8ad334bfe6e7a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } }
I20260928 08:42:17.708339 20740 leader_election.cc:288] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [CANDIDATE]: Term 1 election: Requested vote from peers 3a32a73c9a2a4c0583131315ae1f70f8 (127.0.97.1:46811), fe82770a836c4f5ab0c9503b8014b122 (127.0.97.2:34123)
I20260928 08:42:17.708709 20408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d11f738bdf634ea9965ccac381c17d9f" candidate_uuid: "f36294b184614b46ba8ad334bfe6e7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a32a73c9a2a4c0583131315ae1f70f8"
I20260928 08:42:17.708905 20408 raft_consensus.cc:3037] T d11f738bdf634ea9965ccac381c17d9f P 3a32a73c9a2a4c0583131315ae1f70f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:17.708887 20539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d11f738bdf634ea9965ccac381c17d9f" candidate_uuid: "f36294b184614b46ba8ad334bfe6e7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe82770a836c4f5ab0c9503b8014b122"
I20260928 08:42:17.709088 20539 raft_consensus.cc:3037] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:17.711236 20408 raft_consensus.cc:2454] T d11f738bdf634ea9965ccac381c17d9f P 3a32a73c9a2a4c0583131315ae1f70f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f36294b184614b46ba8ad334bfe6e7a5 in term 1.
I20260928 08:42:17.711567 20539 raft_consensus.cc:2454] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f36294b184614b46ba8ad334bfe6e7a5 in term 1.
I20260928 08:42:17.711603 20604 leader_election.cc:302] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a32a73c9a2a4c0583131315ae1f70f8, f36294b184614b46ba8ad334bfe6e7a5; no voters:
I20260928 08:42:17.711962 20740 raft_consensus.cc:2781] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:17.712392 20740 raft_consensus.cc:686] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [term 1 LEADER]: Becoming Leader. State: Replica: f36294b184614b46ba8ad334bfe6e7a5, State: Running, Role: LEADER
I20260928 08:42:17.712906 20740 consensus_queue.cc:237] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a32a73c9a2a4c0583131315ae1f70f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46811 } } peers { permanent_uuid: "fe82770a836c4f5ab0c9503b8014b122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34123 } } peers { permanent_uuid: "f36294b184614b46ba8ad334bfe6e7a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } }
I20260928 08:42:17.717695 20266 catalog_manager.cc:5495] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f36294b184614b46ba8ad334bfe6e7a5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "f36294b184614b46ba8ad334bfe6e7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a32a73c9a2a4c0583131315ae1f70f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe82770a836c4f5ab0c9503b8014b122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f36294b184614b46ba8ad334bfe6e7a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } health_report { overall_health: HEALTHY } } }
W20260928 08:42:17.747898 20454 tablet.cc:2369] T d11f738bdf634ea9965ccac381c17d9f P 3a32a73c9a2a4c0583131315ae1f70f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:42:17.766649 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:17.761348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34854 (local address 127.0.97.3:39825)
0928 08:42:17.761546 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:17.761552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:17.761572 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:42:17.761779 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:17.761783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:17.761869 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:17.762035 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:42:17.762378 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.763119 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.764512 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.764920 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.765061 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:17.765532 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:42:17.766035 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:42:17.766186 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:17.766266 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 08:42:17.766346 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:42:17.774645 20408 raft_consensus.cc:1261] T d11f738bdf634ea9965ccac381c17d9f P 3a32a73c9a2a4c0583131315ae1f70f8 [term 1 FOLLOWER]: Refusing update from remote peer f36294b184614b46ba8ad334bfe6e7a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:42:17.774645 20539 raft_consensus.cc:1261] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122 [term 1 FOLLOWER]: Refusing update from remote peer f36294b184614b46ba8ad334bfe6e7a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:42:17.775550 20742 consensus_queue.cc:1035] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a32a73c9a2a4c0583131315ae1f70f8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:42:17.775892 20740 consensus_queue.cc:1035] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe82770a836c4f5ab0c9503b8014b122" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:42:17.786976 20745 mvcc.cc:205] Tried to move back new op lower bound from 7334235905114984448 to 7334235904876359680. Current Snapshot: MvccSnapshot[applied={T|T < 7334235905114984448}]
I20260928 08:42:17.802368 20266 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:42:17.802713 20266 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:42:17.806277 20266 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b636956a86d74953b32a61f427329687: Sending DeleteTablet for 3 replicas of tablet d11f738bdf634ea9965ccac381c17d9f
I20260928 08:42:17.807549 20388 tablet_service.cc:1514] Processing DeleteTablet for tablet d11f738bdf634ea9965ccac381c17d9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:42:17 UTC) from {username='slave'} at 127.0.0.1:52874
I20260928 08:42:17.807646 20519 tablet_service.cc:1514] Processing DeleteTablet for tablet d11f738bdf634ea9965ccac381c17d9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:42:17 UTC) from {username='slave'} at 127.0.0.1:40200
I20260928 08:42:17.807729 20649 tablet_service.cc:1514] Processing DeleteTablet for tablet d11f738bdf634ea9965ccac381c17d9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:42:17 UTC) from {username='slave'} at 127.0.0.1:34848
I20260928 08:42:17.808681 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 20326
I20260928 08:42:17.818308 20757 tablet_replica.cc:330] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5: stopping tablet replica
I20260928 08:42:17.818887 20757 raft_consensus.cc:2229] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:42:17.819727 20757 raft_consensus.cc:2258] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:42:17.822190 20757 ts_tablet_manager.cc:1905] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:42:17.827262 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:17.809975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.0.97.1:46811)
0928 08:42:17.810217 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:17.810226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:17.817986 (+ 7760us) server_negotiation.cc:407] Connection header received
0928 08:42:17.818780 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:17.818787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:17.818909 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:17.819003 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:17.819824 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.821217 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.823172 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:17.823817 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:17.823918 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:17.825425 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:17.825437 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:17.825457 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:17.826223 (+ 766us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:17.826294 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:17.826320 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:17.826405 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:17.826709 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 08:42:17.826840 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 08:42:17.828713 20448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:17.829403 20756 tablet_replica.cc:330] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122: stopping tablet replica
I20260928 08:42:17.829880 20756 raft_consensus.cc:2229] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:42:17.830109 20757 ts_tablet_manager.cc:1918] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:42:17.830327 20757 log.cc:1196] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-2/wal/wals/d11f738bdf634ea9965ccac381c17d9f
I20260928 08:42:17.830405 20756 raft_consensus.cc:2258] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:42:17.831007 20757 ts_tablet_manager.cc:1939] T d11f738bdf634ea9965ccac381c17d9f P f36294b184614b46ba8ad334bfe6e7a5: Deleting consensus metadata
I20260928 08:42:17.833324 20254 catalog_manager.cc:4841] TS f36294b184614b46ba8ad334bfe6e7a5 (127.0.97.3:39825): tablet d11f738bdf634ea9965ccac381c17d9f (table test-table [id=b973376a2d474d0897ccaa91317dacf1]) successfully deleted
I20260928 08:42:17.835835 20756 ts_tablet_manager.cc:1905] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:42:17.841360 20756 ts_tablet_manager.cc:1918] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:42:17.841544 20756 log.cc:1196] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122: Deleting WAL directory at /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584618551208-388-0/minicluster-data/ts-1/wal/wals/d11f738bdf634ea9965ccac381c17d9f
I20260928 08:42:17.841996 20756 ts_tablet_manager.cc:1939] T d11f738bdf634ea9965ccac381c17d9f P fe82770a836c4f5ab0c9503b8014b122: Deleting consensus metadata
I20260928 08:42:17.843583 20254 catalog_manager.cc:4841] TS fe82770a836c4f5ab0c9503b8014b122 (127.0.97.2:34123): tablet d11f738bdf634ea9965ccac381c17d9f (table test-table [id=b973376a2d474d0897ccaa91317dacf1]) successfully deleted
I20260928 08:42:17.988041 20755 tablet_replica.cc:330] T d11f738bdf634ea9965ccac381c17d9f P 3a32a73c9a2a4c0583131315ae1f70f8: stopping tablet replica
I20260928 08:42:17.988404 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 20326
I20260928 08:42:18.007297 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 20457
W20260928 08:42:18.007290 20253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:46811 (error 108)
I20260928 08:42:18.009991 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:18.009175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46811 (local address 127.0.0.1:52882)
0928 08:42:18.009630 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:18.009766 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":250,"threads_started":1}
W20260928 08:42:18.010273 20253 catalog_manager.cc:4584] TS 3a32a73c9a2a4c0583131315ae1f70f8 (127.0.97.1:46811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d11f738bdf634ea9965ccac381c17d9f: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46811: connect: Connection refused (error 111)
I20260928 08:42:18.015916 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:18.008177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40202 (local address 127.0.97.2:34123)
0928 08:42:18.008392 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:18.008398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:18.008580 (+ 182us) server_negotiation.cc:407] Connection header received
0928 08:42:18.008893 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:18.008903 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:18.009068 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:18.009182 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:18.009713 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:18.010797 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:18.012592 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:18.012981 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:18.013087 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:18.014241 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:18.014249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:18.014262 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:18.015082 (+ 820us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:18.015158 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:18.015193 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:18.015298 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:18.015447 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 08:42:18.015540 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:42:18.016543 20579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:18.064117 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:18.063566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46811 (local address 127.0.0.1:52896)
0928 08:42:18.063815 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:18.063929 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 08:42:18.065282 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:18.064862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46811 (local address 127.0.0.1:52898)
0928 08:42:18.065041 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:18.065123 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 08:42:18.111626 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 20457
I20260928 08:42:18.125655 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:18.125225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46811 (local address 127.0.0.1:52906)
0928 08:42:18.125423 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:18.125517 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 08:42:18.126720 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:18.126352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46811 (local address 127.0.0.1:52916)
0928 08:42:18.126518 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:18.126591 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 08:42:18.135566 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 20587
I20260928 08:42:18.143126 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:18.136487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.0.97.3:39825)
0928 08:42:18.136646 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:18.136650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:18.136669 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:42:18.136725 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:18.136728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:18.136800 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:18.136885 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:18.137345 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:18.138158 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:18.139995 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:18.140401 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:18.140489 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:18.141656 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:18.141664 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:18.141679 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:18.142298 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:18.142347 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:18.142366 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:18.142456 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:18.142666 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 08:42:18.142757 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 08:42:18.143823 20709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:18.207676 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:18.207073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46811 (local address 127.0.0.1:52920)
0928 08:42:18.207333 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:18.207484 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 08:42:18.208694 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:18.208307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46811 (local address 127.0.0.1:52922)
0928 08:42:18.208495 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:18.208566 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 08:42:18.268064 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 20587
I20260928 08:42:18.287249 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 20236
I20260928 08:42:18.288393 20297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:18.399727 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:18.399064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46811 (local address 127.0.0.1:52938)
0928 08:42:18.399334 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:18.399482 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 08:42:18.399909 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 20236
2026-09-28T08:42:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3013 ms)
[----------] 6 tests from EncryptionPolicyTest (21655 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:18 dist-test-slave-w6c0 krb5kdc[20767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:18 dist-test-slave-w6c0 krb5kdc[20767](info): set up 2 sockets
Sep 28 08:42:18 dist-test-slave-w6c0 krb5kdc[20767](info): commencing operation
krb5kdc: starting...
W20260928 08:42:19.515003 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:19 dist-test-slave-w6c0 krb5kdc[20767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:42:19.758170 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33491
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:20.085355 20780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:20.085692 20780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:20.085857 20780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:20.093169 20780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:20.093318 20780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:20.093415 20780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:20.093518 20780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:20.093602 20780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:20.103504 20780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33491
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:20.104970 20780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:20.106225 20780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:20.118162 20785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:20.118192 20786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:20.118145 20788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:20.118945 20780 server_base.cc:1034] running on GCE node
I20260928 08:42:20.119732 20780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:20.121090 20780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:20.122334 20780 hybrid_clock.cc:648] HybridClock initialized: now 1790584940122279 us; error 47 us; skew 500 ppm
I20260928 08:42:20.122771 20780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:20 dist-test-slave-w6c0 krb5kdc[20767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:20.125897 20780 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:42:20.127296 20780 webserver.cc:460] Webserver started at http://127.0.0.1:42609/ using document root <none> and password file <none>
I20260928 08:42:20.127913 20780 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:20.128018 20780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:20.128258 20780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:20.131201 20780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "fb9077eab1c74020949c94a322f6b622"
format_stamp: "Formatted at 2026-09-28 08:42:20 on dist-test-slave-w6c0"
server_key: "d3be3de11730afa4b8fb2b694c159335"
server_key_iv: "d401d0eb952010996b8e10ef761dcace"
server_key_version: "encryptionkey@0"
I20260928 08:42:20.131949 20780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "fb9077eab1c74020949c94a322f6b622"
format_stamp: "Formatted at 2026-09-28 08:42:20 on dist-test-slave-w6c0"
server_key: "d3be3de11730afa4b8fb2b694c159335"
server_key_iv: "d401d0eb952010996b8e10ef761dcace"
server_key_version: "encryptionkey@0"
I20260928 08:42:20.136883 20780 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:42:20.140174 20795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:20.141376 20780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 08:42:20.141553 20780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "fb9077eab1c74020949c94a322f6b622"
format_stamp: "Formatted at 2026-09-28 08:42:20 on dist-test-slave-w6c0"
server_key: "d3be3de11730afa4b8fb2b694c159335"
server_key_iv: "d401d0eb952010996b8e10ef761dcace"
server_key_version: "encryptionkey@0"
I20260928 08:42:20.141734 20780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:20.177515 20780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:20.181906 20780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:20.182221 20780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:20.200505 20780 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33491
I20260928 08:42:20.200525 20846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33491 every 8 connection(s)
I20260928 08:42:20.201889 20780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:20.205519 20847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:20.210951 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 20780
I20260928 08:42:20.211166 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:20.211761 388 external_mini_cluster.cc:1428] Setting key f99417cb3d1a858e92d10143663fb91f
I20260928 08:42:20.216532 20847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622: Bootstrap starting.
I20260928 08:42:20.220222 20847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:20.221576 20847 log.cc:824] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:20.224686 20847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622: No bootstrap required, opened a new log
Sep 28 08:42:20 dist-test-slave-w6c0 krb5kdc[20767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 08:42:20.233961 20847 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9077eab1c74020949c94a322f6b622" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33491 } }
I20260928 08:42:20.234481 20847 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:20.234640 20847 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb9077eab1c74020949c94a322f6b622, State: Initialized, Role: FOLLOWER
I20260928 08:42:20.235412 20847 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9077eab1c74020949c94a322f6b622" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33491 } }
I20260928 08:42:20.235702 20847 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:20.235843 20847 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:20.236025 20847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:20.238368 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:20.213844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45469 (local address 127.0.0.1:33491)
0928 08:42:20.214798 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:20.214814 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:20.215197 (+ 383us) server_negotiation.cc:407] Connection header received
0928 08:42:20.217595 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:20.217622 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:20.218118 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:20.218597 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:20.220053 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:20.222051 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:20.223627 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:20.224622 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:20.224823 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:20.228861 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:20.228873 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:20.228896 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:20.228938 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:20.232911 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:20.233672 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:20.233681 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:20.233688 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:20.233906 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:20.234564 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:20.234571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:20.234574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:20.235279 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:20.235642 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:20.236967 (+ 1325us) server_negotiation.cc:299] Negotiation successful
0928 08:42:20.237330 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":388,"threads_started":1}
I20260928 08:42:20.238809 20847 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9077eab1c74020949c94a322f6b622" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33491 } }
I20260928 08:42:20.239401 20847 leader_election.cc:302] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb9077eab1c74020949c94a322f6b622; no voters:
I20260928 08:42:20.240002 20847 leader_election.cc:288] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:20.240108 20852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:20.242098 20847 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:20.242111 20852 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622 [term 1 LEADER]: Becoming Leader. State: Replica: fb9077eab1c74020949c94a322f6b622, State: Running, Role: LEADER
I20260928 08:42:20.242900 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:42:20.242785 20852 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9077eab1c74020949c94a322f6b622" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33491 } }
I20260928 08:42:20.245239 20854 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb9077eab1c74020949c94a322f6b622. Latest consensus state: current_term: 1 leader_uuid: "fb9077eab1c74020949c94a322f6b622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9077eab1c74020949c94a322f6b622" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33491 } } }
I20260928 08:42:20.245597 20854 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:20.246243 20853 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb9077eab1c74020949c94a322f6b622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9077eab1c74020949c94a322f6b622" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33491 } } }
I20260928 08:42:20.246579 20853 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:20.256464 20858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:20.262471 20858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:20.280023 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:20.254882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56809 (local address 127.0.0.1:33491)
0928 08:42:20.255107 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:20.255115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:20.258597 (+ 3482us) server_negotiation.cc:407] Connection header received
0928 08:42:20.259650 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:20.259657 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:20.259754 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:20.259831 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:20.261646 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:20.263002 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:20.265425 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:20.266401 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:20.266550 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:20.270715 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:20.270725 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:20.270734 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:20.270786 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:20.275411 (+ 4625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:20.276608 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:20.276617 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:20.276624 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:20.276805 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:20.277535 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:20.277543 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:20.277549 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:20.278065 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:20.278362 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:20.279232 (+ 870us) server_negotiation.cc:299] Negotiation successful
0928 08:42:20.279483 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:42:20.284555 20858 catalog_manager.cc:1347] Generated new cluster ID: dad27ab0b8e3481f8eb71d3a18e1cfcb
I20260928 08:42:20.284657 20858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:20.323251 20858 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:20.324779 20858 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:20.336041 20858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb9077eab1c74020949c94a322f6b622: Generated new TSK 0
I20260928 08:42:20.336844 20858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:20.351208 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 20780
I20260928 08:42:20.352526 20842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:20.449777 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 20780
W20260928 08:42:20.467640 20765 connection.cc:535] client connection to 127.0.0.1:33491 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33491): Connection reset by peer (error 104)
2026-09-28T08:42:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2050 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:20 dist-test-slave-w6c0 krb5kdc[20880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:20 dist-test-slave-w6c0 krb5kdc[20880](info): set up 2 sockets
Sep 28 08:42:20 dist-test-slave-w6c0 krb5kdc[20880](info): commencing operation
krb5kdc: starting...
W20260928 08:42:21.558930 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:21 dist-test-slave-w6c0 krb5kdc[20880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:42:21.812083 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:22.137392 20893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:22.137683 20893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:22.137867 20893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:22.145228 20893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:22.145339 20893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:22.145429 20893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:22.145542 20893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:22.145630 20893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:22.155516 20893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39859
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:22.156878 20893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:22.158278 20893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:22.170221 20899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:22.170251 20901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:22.170341 20898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:22.170881 20893 server_base.cc:1034] running on GCE node
I20260928 08:42:22.171715 20893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:22.173002 20893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:22.174197 20893 hybrid_clock.cc:648] HybridClock initialized: now 1790584942174151 us; error 37 us; skew 500 ppm
I20260928 08:42:22.174605 20893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:22 dist-test-slave-w6c0 krb5kdc[20880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:22.183336 20893 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:42:22.184952 20893 webserver.cc:460] Webserver started at http://127.0.0.1:33169/ using document root <none> and password file <none>
I20260928 08:42:22.185658 20893 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:22.185797 20893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:22.186122 20893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:22.189186 20893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "f2f70f97d16342b2bb806e0c2f82d797"
format_stamp: "Formatted at 2026-09-28 08:42:22 on dist-test-slave-w6c0"
server_key: "aa6a75a6a67f3a56e288790d0d2d3ee2"
server_key_iv: "fa9353ca164af86405a3ac38c4697a3c"
server_key_version: "encryptionkey@0"
I20260928 08:42:22.189960 20893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "f2f70f97d16342b2bb806e0c2f82d797"
format_stamp: "Formatted at 2026-09-28 08:42:22 on dist-test-slave-w6c0"
server_key: "aa6a75a6a67f3a56e288790d0d2d3ee2"
server_key_iv: "fa9353ca164af86405a3ac38c4697a3c"
server_key_version: "encryptionkey@0"
I20260928 08:42:22.195340 20893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 08:42:22.198817 20908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:22.200110 20893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:42:22.200290 20893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "f2f70f97d16342b2bb806e0c2f82d797"
format_stamp: "Formatted at 2026-09-28 08:42:22 on dist-test-slave-w6c0"
server_key: "aa6a75a6a67f3a56e288790d0d2d3ee2"
server_key_iv: "fa9353ca164af86405a3ac38c4697a3c"
server_key_version: "encryptionkey@0"
I20260928 08:42:22.200493 20893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:22.226282 20893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:22.230865 20893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:22.231168 20893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:22.250790 20893 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39859
I20260928 08:42:22.250814 20959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39859 every 8 connection(s)
I20260928 08:42:22.252229 20893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:22.255985 20960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:22.256844 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 20893
I20260928 08:42:22.257063 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:22.257486 388 external_mini_cluster.cc:1428] Setting key 80405f8c8c55107cc8a25327270714c8
I20260928 08:42:22.268474 20960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797: Bootstrap starting.
I20260928 08:42:22.270757 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:22.259289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46047 (local address 127.0.0.1:39859)
0928 08:42:22.260009 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:22.260024 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:22.260066 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:42:22.262216 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:22.262237 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:22.262686 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:22.263062 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:22.263591 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:22.264831 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:22.266209 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:22.266964 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:22.267166 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:22.268441 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:22.268449 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:22.268465 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:22.269039 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:22.269122 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:22.269147 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:22.269316 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:22.269539 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 08:42:22.269707 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":304,"threads_started":1}
I20260928 08:42:22.271901 20960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:22.272980 20960 log.cc:824] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:22.274524 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:42:22.276029 20960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797: No bootstrap required, opened a new log
I20260928 08:42:22.285189 20960 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f70f97d16342b2bb806e0c2f82d797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39859 } }
I20260928 08:42:22.285765 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:22.278935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49271 (local address 127.0.0.1:39859)
0928 08:42:22.279117 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:22.279123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:22.279443 (+ 320us) server_negotiation.cc:407] Connection header received
0928 08:42:22.279584 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:22.279590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:22.279676 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:22.279786 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:22.280239 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:22.281430 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:22.282913 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:22.283663 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:22.283805 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:22.284972 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:22.284978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:22.284983 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:22.285004 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:22.285069 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:22.285100 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:22.285236 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:22.285396 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 08:42:22.285450 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:42:22.285756 20960 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:22.285902 20960 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2f70f97d16342b2bb806e0c2f82d797, State: Initialized, Role: FOLLOWER
I20260928 08:42:22.286635 20960 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f70f97d16342b2bb806e0c2f82d797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39859 } }
I20260928 08:42:22.286927 20960 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:22.287072 20960 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:22.287258 20960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:22.289674 20960 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f70f97d16342b2bb806e0c2f82d797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39859 } }
I20260928 08:42:22.290134 20960 leader_election.cc:302] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2f70f97d16342b2bb806e0c2f82d797; no voters:
I20260928 08:42:22.290712 20960 leader_election.cc:288] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:22.290855 20970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:22.291579 20970 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797 [term 1 LEADER]: Becoming Leader. State: Replica: f2f70f97d16342b2bb806e0c2f82d797, State: Running, Role: LEADER
I20260928 08:42:22.292129 20970 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f70f97d16342b2bb806e0c2f82d797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39859 } }
I20260928 08:42:22.292878 20960 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:22.295301 20971 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2f70f97d16342b2bb806e0c2f82d797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f70f97d16342b2bb806e0c2f82d797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39859 } } }
I20260928 08:42:22.295775 20971 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:22.296577 20972 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2f70f97d16342b2bb806e0c2f82d797. Latest consensus state: current_term: 1 leader_uuid: "f2f70f97d16342b2bb806e0c2f82d797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f70f97d16342b2bb806e0c2f82d797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39859 } } }
I20260928 08:42:22.297048 20972 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:22.297412 20977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:22.302830 20977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:22.312150 20977 catalog_manager.cc:1347] Generated new cluster ID: f23af4948ace40a9af3444cdf61b597e
I20260928 08:42:22.312281 20977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:22.350121 20977 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:22.351495 20977 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:22.359813 20977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2f70f97d16342b2bb806e0c2f82d797: Generated new TSK 0
I20260928 08:42:22.360625 20977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:22.375985 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 20893
W20260928 08:42:22.377000 20955 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46047
W20260928 08:42:22.377686 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 20893: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:42:22.377851 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 20893
W20260928 08:42:22.394799 20878 connection.cc:535] client connection to 127.0.0.1:39859 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39859): Connection reset by peer (error 104)
2026-09-28T08:42:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1924 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:22 dist-test-slave-w6c0 krb5kdc[20991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:22 dist-test-slave-w6c0 krb5kdc[20991](info): set up 2 sockets
Sep 28 08:42:22 dist-test-slave-w6c0 krb5kdc[20991](info): commencing operation
krb5kdc: starting...
W20260928 08:42:23.483489 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:23 dist-test-slave-w6c0 krb5kdc[20991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:42:23.735698 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41813
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:24.061780 21004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:24.062052 21004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:24.062217 21004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:24.069509 21004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:24.069612 21004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:24.069742 21004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:24.069847 21004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:24.069931 21004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:24.080260 21004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41813
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:24.081578 21004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:24.082870 21004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:24.094559 21010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:24.094550 21012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:24.094550 21009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:24.095433 21004 server_base.cc:1034] running on GCE node
I20260928 08:42:24.096110 21004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:24.097452 21004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:24.098685 21004 hybrid_clock.cc:648] HybridClock initialized: now 1790584944098644 us; error 37 us; skew 500 ppm
I20260928 08:42:24.099051 21004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:24 dist-test-slave-w6c0 krb5kdc[20991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:24.103144 21004 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:42:24.105078 21004 webserver.cc:460] Webserver started at http://127.0.0.1:40679/ using document root <none> and password file <none>
I20260928 08:42:24.105798 21004 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:24.105954 21004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:24.106269 21004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:24.109342 21004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "9376973202354ac79543468b77f7b235"
format_stamp: "Formatted at 2026-09-28 08:42:24 on dist-test-slave-w6c0"
server_key: "24a87255775934292f239e04afc310e2"
server_key_iv: "42e29f7a9177875e8fe1873becd9ae04"
server_key_version: "encryptionkey@0"
I20260928 08:42:24.110121 21004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "9376973202354ac79543468b77f7b235"
format_stamp: "Formatted at 2026-09-28 08:42:24 on dist-test-slave-w6c0"
server_key: "24a87255775934292f239e04afc310e2"
server_key_iv: "42e29f7a9177875e8fe1873becd9ae04"
server_key_version: "encryptionkey@0"
I20260928 08:42:24.115404 21004 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 08:42:24.118789 21019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:24.120087 21004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:42:24.120294 21004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "9376973202354ac79543468b77f7b235"
format_stamp: "Formatted at 2026-09-28 08:42:24 on dist-test-slave-w6c0"
server_key: "24a87255775934292f239e04afc310e2"
server_key_iv: "42e29f7a9177875e8fe1873becd9ae04"
server_key_version: "encryptionkey@0"
I20260928 08:42:24.120499 21004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:24.126261 21004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:24.130123 21004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:24.130414 21004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:24.148595 21004 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41813
I20260928 08:42:24.148644 21070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41813 every 8 connection(s)
I20260928 08:42:24.149978 21004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:24.153503 21071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:24.158814 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 21004
I20260928 08:42:24.158988 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:24.159518 388 external_mini_cluster.cc:1428] Setting key 0e82587f5d731e030509b42e85e93ac8
I20260928 08:42:24.164598 21071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235: Bootstrap starting.
I20260928 08:42:24.167344 21074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:24.161323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36857 (local address 127.0.0.1:41813)
0928 08:42:24.162164 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:24.162175 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:24.162213 (+ 38us) server_negotiation.cc:407] Connection header received
0928 08:42:24.164060 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:24.164083 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:24.164395 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:24.164781 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:24.165783 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:24.165789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:24.165801 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:24.165818 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:24.165871 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:24.165889 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:24.166144 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:24.166282 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 08:42:24.166447 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":427,"threads_started":1}
I20260928 08:42:24.167994 21071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:24.169094 21071 log.cc:824] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:24.170763 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:42:24.172031 21071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235: No bootstrap required, opened a new log
I20260928 08:42:24.177615 21074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:24.175112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35013 (local address 127.0.0.1:41813)
0928 08:42:24.175359 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:24.175365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:24.175699 (+ 334us) server_negotiation.cc:407] Connection header received
0928 08:42:24.175824 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:24.175828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:24.175873 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:24.175968 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:24.176931 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:24.176939 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:24.176942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:24.176957 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:24.176996 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:24.177010 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:24.177115 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:24.177242 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 08:42:24.177291 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:42:24.180526 21071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9376973202354ac79543468b77f7b235" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41813 } }
I20260928 08:42:24.180943 21071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:24.181057 21071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9376973202354ac79543468b77f7b235, State: Initialized, Role: FOLLOWER
I20260928 08:42:24.181612 21071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9376973202354ac79543468b77f7b235" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41813 } }
I20260928 08:42:24.181864 21071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:24.182004 21071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:24.182192 21071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:24.184448 21071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9376973202354ac79543468b77f7b235" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41813 } }
I20260928 08:42:24.184942 21071 leader_election.cc:302] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9376973202354ac79543468b77f7b235; no voters:
I20260928 08:42:24.185550 21071 leader_election.cc:288] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:24.185681 21081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:24.186371 21081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235 [term 1 LEADER]: Becoming Leader. State: Replica: 9376973202354ac79543468b77f7b235, State: Running, Role: LEADER
I20260928 08:42:24.186950 21081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9376973202354ac79543468b77f7b235" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41813 } }
I20260928 08:42:24.187662 21071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:24.189616 21082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9376973202354ac79543468b77f7b235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9376973202354ac79543468b77f7b235" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41813 } } }
I20260928 08:42:24.190058 21082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:24.190961 21083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9376973202354ac79543468b77f7b235. Latest consensus state: current_term: 1 leader_uuid: "9376973202354ac79543468b77f7b235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9376973202354ac79543468b77f7b235" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41813 } } }
I20260928 08:42:24.191293 21083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:24.194638 21089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:24.200066 21089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:24.208995 21089 catalog_manager.cc:1347] Generated new cluster ID: e33d3c3024a849f0a58f628ac9aefb16
I20260928 08:42:24.209113 21089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:24.225992 21089 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:24.226420 21089 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:24.233268 21089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9376973202354ac79543468b77f7b235: Generated new TSK 0
I20260928 08:42:24.234520 21089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:24.248211 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 21004
W20260928 08:42:24.249982 21066 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36857
W20260928 08:42:24.250821 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 21004: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:42:24.250977 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 21004
2026-09-28T08:42:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1873 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:24 dist-test-slave-w6c0 krb5kdc[21102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:24 dist-test-slave-w6c0 krb5kdc[21102](info): set up 2 sockets
Sep 28 08:42:24 dist-test-slave-w6c0 krb5kdc[21102](info): commencing operation
krb5kdc: starting...
W20260928 08:42:25.362944 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:25 dist-test-slave-w6c0 krb5kdc[21102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:42:25.615140 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:25.938740 21115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:25.939036 21115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:25.939229 21115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:25.946455 21115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:25.946647 21115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:25.946746 21115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:25.946846 21115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:25.946931 21115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:25.956723 21115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:25.958038 21115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:25.959336 21115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:25.970953 21120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:25.970999 21121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:25.971032 21123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:25.971587 21115 server_base.cc:1034] running on GCE node
I20260928 08:42:25.972301 21115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:25.973569 21115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:25.974763 21115 hybrid_clock.cc:648] HybridClock initialized: now 1790584945974704 us; error 55 us; skew 500 ppm
I20260928 08:42:25.975118 21115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:25 dist-test-slave-w6c0 krb5kdc[21102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:25.978220 21115 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:42:25.979624 21115 webserver.cc:460] Webserver started at http://127.0.0.1:37641/ using document root <none> and password file <none>
I20260928 08:42:25.980283 21115 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:25.980409 21115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:25.980778 21115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:25.983664 21115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "ef951f3a753042368fab0fbf8920f9b1"
format_stamp: "Formatted at 2026-09-28 08:42:25 on dist-test-slave-w6c0"
server_key: "94c6df954dde86b9d06c317567d7728b"
server_key_iv: "6f76401d79281916e219f8606fd9581c"
server_key_version: "encryptionkey@0"
I20260928 08:42:25.984398 21115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "ef951f3a753042368fab0fbf8920f9b1"
format_stamp: "Formatted at 2026-09-28 08:42:25 on dist-test-slave-w6c0"
server_key: "94c6df954dde86b9d06c317567d7728b"
server_key_iv: "6f76401d79281916e219f8606fd9581c"
server_key_version: "encryptionkey@0"
I20260928 08:42:25.989580 21115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 08:42:25.992885 21130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:25.994129 21115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:42:25.994305 21115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "ef951f3a753042368fab0fbf8920f9b1"
format_stamp: "Formatted at 2026-09-28 08:42:25 on dist-test-slave-w6c0"
server_key: "94c6df954dde86b9d06c317567d7728b"
server_key_iv: "6f76401d79281916e219f8606fd9581c"
server_key_version: "encryptionkey@0"
I20260928 08:42:25.994498 21115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:26.006107 21115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:26.010429 21115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:26.010744 21115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:26.029248 21115 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40369
I20260928 08:42:26.029289 21181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40369 every 8 connection(s)
I20260928 08:42:26.030673 21115 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:26.034207 21182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:26.038888 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 21115
I20260928 08:42:26.039086 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:26.039502 388 external_mini_cluster.cc:1428] Setting key beecf5bf67f4ac93fa461b5f4dfd58a1
I20260928 08:42:26.046423 21182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1: Bootstrap starting.
I20260928 08:42:26.050081 21182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:26.051324 21182 log.cc:824] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:26.054126 21182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1: No bootstrap required, opened a new log
Sep 28 08:42:26 dist-test-slave-w6c0 krb5kdc[21102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 08:42:26.063758 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:26.041315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56607 (local address 127.0.0.1:40369)
0928 08:42:26.042183 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:26.042198 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:26.042256 (+ 58us) server_negotiation.cc:407] Connection header received
0928 08:42:26.044808 (+ 2552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:26.044835 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:26.045325 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:26.045735 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:26.047150 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:26.049169 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:26.050660 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:26.051683 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:26.051906 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:26.056528 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:26.056537 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:26.056551 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:26.056587 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:26.059783 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:26.060627 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:26.060632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:26.060637 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:26.060795 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:26.061205 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:26.061208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:26.061210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:26.061714 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:26.061953 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:26.062387 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 08:42:26.062810 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":369,"threads_started":1}
I20260928 08:42:26.063668 21182 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef951f3a753042368fab0fbf8920f9b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40369 } }
I20260928 08:42:26.064110 21182 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:26.064254 21182 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef951f3a753042368fab0fbf8920f9b1, State: Initialized, Role: FOLLOWER
I20260928 08:42:26.064791 21182 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef951f3a753042368fab0fbf8920f9b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40369 } }
I20260928 08:42:26.065039 21182 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:26.065200 21182 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:26.065414 21182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:26.067771 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:42:26.068431 21182 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef951f3a753042368fab0fbf8920f9b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40369 } }
I20260928 08:42:26.068904 21182 leader_election.cc:302] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef951f3a753042368fab0fbf8920f9b1; no voters:
I20260928 08:42:26.069523 21182 leader_election.cc:288] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:26.069634 21187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:26.071971 21182 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:26.076632 21187 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1 [term 1 LEADER]: Becoming Leader. State: Replica: ef951f3a753042368fab0fbf8920f9b1, State: Running, Role: LEADER
I20260928 08:42:26.077345 21187 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef951f3a753042368fab0fbf8920f9b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40369 } }
I20260928 08:42:26.080873 21188 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef951f3a753042368fab0fbf8920f9b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef951f3a753042368fab0fbf8920f9b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40369 } } }
I20260928 08:42:26.081423 21188 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:26.080953 21189 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef951f3a753042368fab0fbf8920f9b1. Latest consensus state: current_term: 1 leader_uuid: "ef951f3a753042368fab0fbf8920f9b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef951f3a753042368fab0fbf8920f9b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40369 } } }
I20260928 08:42:26.082058 21189 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:26.082759 21203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:26.102330 21203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:26.106835 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:26.080462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54437 (local address 127.0.0.1:40369)
0928 08:42:26.080666 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:26.080672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:26.081915 (+ 1243us) server_negotiation.cc:407] Connection header received
0928 08:42:26.082320 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:26.082329 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:26.082453 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:26.082607 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:26.085319 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:26.086845 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:26.089322 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:26.090299 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:26.090427 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:26.094727 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:26.094737 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:26.094746 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:26.094805 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:26.100063 (+ 5258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:26.103579 (+ 3516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:26.103588 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:26.103595 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:26.103813 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:26.104393 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:26.104398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:26.104401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:26.104927 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:26.105213 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:26.105974 (+ 761us) server_negotiation.cc:299] Negotiation successful
0928 08:42:26.106292 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 08:42:26.119575 21203 catalog_manager.cc:1347] Generated new cluster ID: ce4f2a94fd4a41c189e32023bfa7d9a7
I20260928 08:42:26.119717 21203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:26.138932 21203 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:26.140329 21203 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:26.147615 21203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef951f3a753042368fab0fbf8920f9b1: Generated new TSK 0
I20260928 08:42:26.148415 21203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:26.156275 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 21115
I20260928 08:42:26.157374 21177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:26.250090 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 21115
2026-09-28T08:42:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1996 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:26 dist-test-slave-w6c0 krb5kdc[21214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:26 dist-test-slave-w6c0 krb5kdc[21214](info): set up 2 sockets
Sep 28 08:42:26 dist-test-slave-w6c0 krb5kdc[21214](info): commencing operation
krb5kdc: starting...
W20260928 08:42:27.359019 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:27 dist-test-slave-w6c0 krb5kdc[21214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:42:27.621860 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:27.955008 21227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:27.955322 21227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:27.955544 21227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:27.962984 21227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:27.963101 21227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:27.963225 21227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:27.963346 21227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:27.963429 21227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:27.973450 21227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:27.974841 21227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:27.976210 21227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:27.988304 21235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:27.988406 21232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:27.988381 21233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:27.989142 21227 server_base.cc:1034] running on GCE node
I20260928 08:42:27.989940 21227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:27.991458 21227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:27.992669 21227 hybrid_clock.cc:648] HybridClock initialized: now 1790584947992629 us; error 30 us; skew 500 ppm
I20260928 08:42:27.993058 21227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:27 dist-test-slave-w6c0 krb5kdc[21214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:28.002674 21227 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:42:28.004312 21227 webserver.cc:460] Webserver started at http://127.0.0.1:41193/ using document root <none> and password file <none>
I20260928 08:42:28.005064 21227 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:28.005216 21227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:28.005544 21227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:28.008713 21227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "63de4859e16c4d92a53395a7c564a073"
format_stamp: "Formatted at 2026-09-28 08:42:28 on dist-test-slave-w6c0"
server_key: "29f108f00a09943c0a55d1320b78f738"
server_key_iv: "3c5f90e910a93d82082f2ff8dd7b7061"
server_key_version: "encryptionkey@0"
I20260928 08:42:28.009519 21227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "63de4859e16c4d92a53395a7c564a073"
format_stamp: "Formatted at 2026-09-28 08:42:28 on dist-test-slave-w6c0"
server_key: "29f108f00a09943c0a55d1320b78f738"
server_key_iv: "3c5f90e910a93d82082f2ff8dd7b7061"
server_key_version: "encryptionkey@0"
I20260928 08:42:28.014753 21227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:42:28.018178 21242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:28.019613 21227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:42:28.019806 21227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "63de4859e16c4d92a53395a7c564a073"
format_stamp: "Formatted at 2026-09-28 08:42:28 on dist-test-slave-w6c0"
server_key: "29f108f00a09943c0a55d1320b78f738"
server_key_iv: "3c5f90e910a93d82082f2ff8dd7b7061"
server_key_version: "encryptionkey@0"
I20260928 08:42:28.020018 21227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:28.045974 21227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:28.050191 21227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:28.050494 21227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:28.069283 21227 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45427
I20260928 08:42:28.069322 21293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45427 every 8 connection(s)
I20260928 08:42:28.070741 21227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:28.074376 21294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:28.077508 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 21227
I20260928 08:42:28.077690 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:28.078212 388 external_mini_cluster.cc:1428] Setting key 03db22da2023be16207ffb182152dd12
I20260928 08:42:28.086623 21294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073: Bootstrap starting.
I20260928 08:42:28.090324 21294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:28.091464 21294 log.cc:824] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:28.091579 21297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:28.080080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49879 (local address 127.0.0.1:45427)
0928 08:42:28.080933 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:28.080945 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:28.080986 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:42:28.082720 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:28.082742 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:28.083141 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:28.083465 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:28.084059 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:28.085433 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:28.087264 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:28.087914 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:28.088059 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:28.089303 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:28.089310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:28.089325 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:28.089862 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:28.089933 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:28.089957 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:28.090053 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:28.090297 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 08:42:28.090595 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":393,"threads_started":1}
I20260928 08:42:28.094568 21294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073: No bootstrap required, opened a new log
I20260928 08:42:28.095237 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:42:28.104140 21294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63de4859e16c4d92a53395a7c564a073" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } }
I20260928 08:42:28.104621 21294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:28.104733 21294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63de4859e16c4d92a53395a7c564a073, State: Initialized, Role: FOLLOWER
I20260928 08:42:28.105419 21294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63de4859e16c4d92a53395a7c564a073" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } }
I20260928 08:42:28.105754 21294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:28.105949 21294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:28.106156 21294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:28.106549 21297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:28.099614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47101 (local address 127.0.0.1:45427)
0928 08:42:28.099815 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:28.099826 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:28.100365 (+ 539us) server_negotiation.cc:407] Connection header received
0928 08:42:28.100538 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:28.100548 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:28.100617 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:28.100706 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:28.101364 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:28.102103 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:28.103963 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:28.104377 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:28.104488 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:28.105662 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:28.105688 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:28.105695 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:28.105721 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:28.105783 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:28.105801 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:28.105919 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:28.106122 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 08:42:28.106212 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:42:28.109071 21294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63de4859e16c4d92a53395a7c564a073" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } }
I20260928 08:42:28.109586 21294 leader_election.cc:302] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63de4859e16c4d92a53395a7c564a073; no voters:
I20260928 08:42:28.110144 21294 leader_election.cc:288] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:28.110311 21304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:28.111076 21304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073 [term 1 LEADER]: Becoming Leader. State: Replica: 63de4859e16c4d92a53395a7c564a073, State: Running, Role: LEADER
I20260928 08:42:28.111617 21304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63de4859e16c4d92a53395a7c564a073" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } }
I20260928 08:42:28.112295 21294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:28.115880 21306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63de4859e16c4d92a53395a7c564a073. Latest consensus state: current_term: 1 leader_uuid: "63de4859e16c4d92a53395a7c564a073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63de4859e16c4d92a53395a7c564a073" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } } }
I20260928 08:42:28.116397 21306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:28.116211 21305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63de4859e16c4d92a53395a7c564a073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63de4859e16c4d92a53395a7c564a073" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } } }
I20260928 08:42:28.116561 21305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:28.117450 21314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:28.121924 21314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:28.130779 21314 catalog_manager.cc:1347] Generated new cluster ID: 4ce51bda53584dbc98f5bd2399a76481
I20260928 08:42:28.130890 21314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:28.139029 21314 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:28.140769 21314 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:28.154098 21314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63de4859e16c4d92a53395a7c564a073: Generated new TSK 0
I20260928 08:42:28.155117 21314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:28.166754 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 21227
W20260928 08:42:28.167848 21289 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49879
W20260928 08:42:28.168553 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 21227: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:42:28.168704 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 21227
2026-09-28T08:42:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1919 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:28 dist-test-slave-w6c0 krb5kdc[21325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:28 dist-test-slave-w6c0 krb5kdc[21325](info): set up 2 sockets
Sep 28 08:42:28 dist-test-slave-w6c0 krb5kdc[21325](info): commencing operation
krb5kdc: starting...
W20260928 08:42:29.299036 388 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.012s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:29 dist-test-slave-w6c0 krb5kdc[21325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:42:29.568136 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34337
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:29.901139 21338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:29.901435 21338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:29.901598 21338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:29.908979 21338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:29.909099 21338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:29.909188 21338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:29.909302 21338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:29.909389 21338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:29.919440 21338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:29.920809 21338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:29.922168 21338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:29.934477 21343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:29.934530 21346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:29.934584 21344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:29.935497 21338 server_base.cc:1034] running on GCE node
I20260928 08:42:29.936499 21338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:29.938053 21338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:29.939304 21338 hybrid_clock.cc:648] HybridClock initialized: now 1790584949939263 us; error 62 us; skew 500 ppm
I20260928 08:42:29.939688 21338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:29 dist-test-slave-w6c0 krb5kdc[21325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:29.943156 21338 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:42:29.944722 21338 webserver.cc:460] Webserver started at http://127.0.0.1:41329/ using document root <none> and password file <none>
I20260928 08:42:29.945446 21338 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:29.945591 21338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:29.945925 21338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:29.949055 21338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "8c6f2a77cd6f4fa9b26bf8ab596a190b"
format_stamp: "Formatted at 2026-09-28 08:42:29 on dist-test-slave-w6c0"
server_key: "f572fe7432c035a3463b7d01259bc960"
server_key_iv: "3a2bdc85fe6c2123b87ec3c7db790fca"
server_key_version: "encryptionkey@0"
I20260928 08:42:29.949883 21338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "8c6f2a77cd6f4fa9b26bf8ab596a190b"
format_stamp: "Formatted at 2026-09-28 08:42:29 on dist-test-slave-w6c0"
server_key: "f572fe7432c035a3463b7d01259bc960"
server_key_iv: "3a2bdc85fe6c2123b87ec3c7db790fca"
server_key_version: "encryptionkey@0"
I20260928 08:42:29.955190 21338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260928 08:42:29.958761 21353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:29.960122 21338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:42:29.960311 21338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "8c6f2a77cd6f4fa9b26bf8ab596a190b"
format_stamp: "Formatted at 2026-09-28 08:42:29 on dist-test-slave-w6c0"
server_key: "f572fe7432c035a3463b7d01259bc960"
server_key_iv: "3a2bdc85fe6c2123b87ec3c7db790fca"
server_key_version: "encryptionkey@0"
I20260928 08:42:29.960517 21338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:29.966384 21338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:29.971120 21338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:29.971429 21338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:29.990381 21338 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34337
I20260928 08:42:29.990417 21404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34337 every 8 connection(s)
I20260928 08:42:29.991837 21338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:29.993893 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 21338
I20260928 08:42:29.994100 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:29.994626 388 external_mini_cluster.cc:1428] Setting key df58d45e18ea1f896c11572b0fb1e34a
I20260928 08:42:29.996045 21405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:30.002274 21408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:29.996406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39853 (local address 127.0.0.1:34337)
0928 08:42:29.997185 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:29.997197 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:29.997237 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:42:29.998913 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:29.998934 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:29.999273 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:29.999602 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:30.000715 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:30.000721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:30.000732 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:30.000746 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:30.000836 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:30.000860 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:30.001037 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:30.001133 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 08:42:30.001348 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":372,"threads_started":1}
I20260928 08:42:30.005903 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:42:30.008725 21405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b: Bootstrap starting.
I20260928 08:42:30.012585 21405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:30.012676 21408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:30.010106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34553 (local address 127.0.0.1:34337)
0928 08:42:30.010304 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:30.010310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:30.010842 (+ 532us) server_negotiation.cc:407] Connection header received
0928 08:42:30.011006 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:30.011010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:30.011058 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:30.011136 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:30.012027 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:30.012032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:30.012035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:30.012049 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:30.012084 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:30.012098 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:30.012182 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:30.012326 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 08:42:30.012371 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:42:30.013703 21405 log.cc:824] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:30.016324 21405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b: No bootstrap required, opened a new log
I20260928 08:42:30.023518 21405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6f2a77cd6f4fa9b26bf8ab596a190b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34337 } }
I20260928 08:42:30.023905 21405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:30.024010 21405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c6f2a77cd6f4fa9b26bf8ab596a190b, State: Initialized, Role: FOLLOWER
I20260928 08:42:30.024616 21405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6f2a77cd6f4fa9b26bf8ab596a190b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34337 } }
I20260928 08:42:30.024854 21405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:30.024997 21405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:30.025174 21405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:30.028306 21405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6f2a77cd6f4fa9b26bf8ab596a190b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34337 } }
I20260928 08:42:30.028916 21405 leader_election.cc:302] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c6f2a77cd6f4fa9b26bf8ab596a190b; no voters:
I20260928 08:42:30.029524 21405 leader_election.cc:288] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:30.029690 21415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:30.030367 21415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b [term 1 LEADER]: Becoming Leader. State: Replica: 8c6f2a77cd6f4fa9b26bf8ab596a190b, State: Running, Role: LEADER
I20260928 08:42:30.030923 21415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6f2a77cd6f4fa9b26bf8ab596a190b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34337 } }
I20260928 08:42:30.031646 21405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:30.033823 21417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c6f2a77cd6f4fa9b26bf8ab596a190b. Latest consensus state: current_term: 1 leader_uuid: "8c6f2a77cd6f4fa9b26bf8ab596a190b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6f2a77cd6f4fa9b26bf8ab596a190b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34337 } } }
I20260928 08:42:30.033648 21416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c6f2a77cd6f4fa9b26bf8ab596a190b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6f2a77cd6f4fa9b26bf8ab596a190b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34337 } } }
I20260928 08:42:30.034121 21417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:30.034173 21416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:30.036419 21420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:30.041610 21420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:30.050868 21420 catalog_manager.cc:1347] Generated new cluster ID: 17d9df1ac3c44137bd0e1f9b6235c9eb
I20260928 08:42:30.050994 21420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:30.060951 21420 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:30.061380 21420 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:30.068049 21420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c6f2a77cd6f4fa9b26bf8ab596a190b: Generated new TSK 0
I20260928 08:42:30.068881 21420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:30.075168 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 21338
W20260928 08:42:30.076614 21400 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39853
W20260928 08:42:30.077416 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 21338: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:42:30.077579 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 21338
2026-09-28T08:42:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1906 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:30 dist-test-slave-w6c0 krb5kdc[21436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:30 dist-test-slave-w6c0 krb5kdc[21436](info): set up 2 sockets
Sep 28 08:42:30 dist-test-slave-w6c0 krb5kdc[21436](info): commencing operation
krb5kdc: starting...
W20260928 08:42:31.186959 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:31 dist-test-slave-w6c0 krb5kdc[21436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:42:31.445750 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43415
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:31.773492 21449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:31.773833 21449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:31.774008 21449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:31.781280 21449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:31.781443 21449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:31.781554 21449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:31.781665 21449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:31.781747 21449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:31.791710 21449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:31.793062 21449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:31.794369 21449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:31.806377 21455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:31.806452 21454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:31.806452 21457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:31.807329 21449 server_base.cc:1034] running on GCE node
I20260928 08:42:31.808022 21449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:31.809445 21449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:31.810602 21449 hybrid_clock.cc:648] HybridClock initialized: now 1790584951810561 us; error 39 us; skew 500 ppm
I20260928 08:42:31.810945 21449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:31 dist-test-slave-w6c0 krb5kdc[21436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:31.814018 21449 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:42:31.815544 21449 webserver.cc:460] Webserver started at http://127.0.0.1:42773/ using document root <none> and password file <none>
I20260928 08:42:31.816185 21449 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:31.816283 21449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:31.816530 21449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:31.819458 21449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "1b7d546e7e5f4d6ca4a0986bd3b87faa"
format_stamp: "Formatted at 2026-09-28 08:42:31 on dist-test-slave-w6c0"
server_key: "7bdf941c7cf660dd83d97c545b5c6fa3"
server_key_iv: "b4cdf1d81ad0abdf77392817c6d6b33f"
server_key_version: "encryptionkey@0"
I20260928 08:42:31.820179 21449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "1b7d546e7e5f4d6ca4a0986bd3b87faa"
format_stamp: "Formatted at 2026-09-28 08:42:31 on dist-test-slave-w6c0"
server_key: "7bdf941c7cf660dd83d97c545b5c6fa3"
server_key_iv: "b4cdf1d81ad0abdf77392817c6d6b33f"
server_key_version: "encryptionkey@0"
I20260928 08:42:31.825217 21449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 08:42:31.828640 21464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:31.829903 21449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 08:42:31.830096 21449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "1b7d546e7e5f4d6ca4a0986bd3b87faa"
format_stamp: "Formatted at 2026-09-28 08:42:31 on dist-test-slave-w6c0"
server_key: "7bdf941c7cf660dd83d97c545b5c6fa3"
server_key_iv: "b4cdf1d81ad0abdf77392817c6d6b33f"
server_key_version: "encryptionkey@0"
I20260928 08:42:31.830302 21449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:31.859499 21449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:31.863948 21449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:31.864251 21449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:31.882485 21449 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43415
I20260928 08:42:31.882601 21515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43415 every 8 connection(s)
I20260928 08:42:31.883834 21449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:31.887423 21516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:31.889642 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 21449
I20260928 08:42:31.889827 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:31.890252 388 external_mini_cluster.cc:1428] Setting key 51f5be3656dc4af7a9f3567e71764589
I20260928 08:42:31.900496 21516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa: Bootstrap starting.
I20260928 08:42:31.903944 21516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:31.904912 21516 log.cc:824] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa: Log is configured to *not* fsync() on all Append() calls
Sep 28 08:42:31 dist-test-slave-w6c0 krb5kdc[21436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 08:42:31.908478 21516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa: No bootstrap required, opened a new log
I20260928 08:42:31.915149 21519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:31.892060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:47007 (local address 127.0.0.1:43415)
0928 08:42:31.892853 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:31.892867 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:31.892917 (+ 50us) server_negotiation.cc:407] Connection header received
0928 08:42:31.895302 (+ 2385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:31.895331 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:31.895824 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:31.896236 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:31.897597 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:31.899611 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:31.900931 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:31.901893 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:31.902117 (+ 224us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:31.906573 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:31.906584 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:31.906602 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:31.906652 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:31.910520 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:31.911328 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:31.911341 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:31.911353 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:31.911561 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:31.912018 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:31.912023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:31.912027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:31.912740 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:31.913063 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:31.913419 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 08:42:31.913759 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":297,"threads_started":1}
I20260928 08:42:31.919081 21516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7d546e7e5f4d6ca4a0986bd3b87faa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43415 } }
I20260928 08:42:31.919523 21516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:31.919631 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:42:31.919661 21516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b7d546e7e5f4d6ca4a0986bd3b87faa, State: Initialized, Role: FOLLOWER
I20260928 08:42:31.920275 21516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7d546e7e5f4d6ca4a0986bd3b87faa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43415 } }
I20260928 08:42:31.920516 21516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:31.920670 21516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:31.920849 21516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:31.923225 21516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7d546e7e5f4d6ca4a0986bd3b87faa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43415 } }
I20260928 08:42:31.923718 21516 leader_election.cc:302] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b7d546e7e5f4d6ca4a0986bd3b87faa; no voters:
I20260928 08:42:31.925211 21516 leader_election.cc:288] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:31.925316 21525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:31.926654 21525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa [term 1 LEADER]: Becoming Leader. State: Replica: 1b7d546e7e5f4d6ca4a0986bd3b87faa, State: Running, Role: LEADER
I20260928 08:42:31.927301 21525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7d546e7e5f4d6ca4a0986bd3b87faa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43415 } }
I20260928 08:42:31.927557 21516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:31.929919 21527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b7d546e7e5f4d6ca4a0986bd3b87faa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7d546e7e5f4d6ca4a0986bd3b87faa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43415 } } }
I20260928 08:42:31.930298 21527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:31.931218 21528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b7d546e7e5f4d6ca4a0986bd3b87faa. Latest consensus state: current_term: 1 leader_uuid: "1b7d546e7e5f4d6ca4a0986bd3b87faa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7d546e7e5f4d6ca4a0986bd3b87faa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43415 } } }
I20260928 08:42:31.932158 21528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:31.934288 21531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:31.939980 21531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:31.948001 21519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:31.924334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:37797 (local address 127.0.0.1:43415)
0928 08:42:31.924560 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:31.924568 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:31.924852 (+ 284us) server_negotiation.cc:407] Connection header received
0928 08:42:31.924991 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:31.924999 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:31.925099 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:31.925229 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:31.927346 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:31.928607 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:31.931099 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:31.932085 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:31.932250 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:31.936669 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:31.936677 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:31.936683 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:31.936726 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:31.942361 (+ 5635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:31.943719 (+ 1358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:31.943728 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:31.943734 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:31.943915 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:31.944588 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:31.944597 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:31.944603 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:31.945210 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:31.945570 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:31.947223 (+ 1653us) server_negotiation.cc:299] Negotiation successful
0928 08:42:31.947473 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:42:31.953142 21531 catalog_manager.cc:1347] Generated new cluster ID: 7756b5e928894e06b6fb84fd85f4e0bb
I20260928 08:42:31.953255 21531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:31.977365 21531 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:31.979298 21531 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:31.987512 21531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b7d546e7e5f4d6ca4a0986bd3b87faa: Generated new TSK 0
I20260928 08:42:31.988562 21531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:32.006320 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 21449
I20260928 08:42:32.007505 21511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:32.101604 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 21449
W20260928 08:42:32.119261 21434 connection.cc:535] client connection to 127.0.0.1:43415 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43415): Connection reset by peer (error 104)
2026-09-28T08:42:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2026 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:32 dist-test-slave-w6c0 krb5kdc[21548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:32 dist-test-slave-w6c0 krb5kdc[21548](info): set up 2 sockets
Sep 28 08:42:32 dist-test-slave-w6c0 krb5kdc[21548](info): commencing operation
krb5kdc: starting...
W20260928 08:42:33.200563 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:33 dist-test-slave-w6c0 krb5kdc[21548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:42:33.462139 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41231
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:33.786676 21561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:33.786962 21561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:33.787124 21561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:33.794217 21561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:33.794334 21561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:33.794443 21561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:33.794585 21561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:33.794680 21561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:33.805382 21561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:33.806838 21561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:33.808138 21561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:33.820034 21569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:33.820101 21566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:33.820080 21567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:33.820798 21561 server_base.cc:1034] running on GCE node
I20260928 08:42:33.821581 21561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:33.823026 21561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:33.824262 21561 hybrid_clock.cc:648] HybridClock initialized: now 1790584953824223 us; error 47 us; skew 500 ppm
I20260928 08:42:33.824610 21561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:33 dist-test-slave-w6c0 krb5kdc[21548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:33.843535 21561 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:42:33.845037 21561 webserver.cc:460] Webserver started at http://127.0.0.1:40049/ using document root <none> and password file <none>
I20260928 08:42:33.845705 21561 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:33.845804 21561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:33.846079 21561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:33.849099 21561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "fd106f2a0a714f129b01112003e7cbbb"
format_stamp: "Formatted at 2026-09-28 08:42:33 on dist-test-slave-w6c0"
server_key: "0cd7f917cf1b3407957cc0894007cad1"
server_key_iv: "1bf594895de9bf867e913dd81e5aca89"
server_key_version: "encryptionkey@0"
I20260928 08:42:33.849817 21561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "fd106f2a0a714f129b01112003e7cbbb"
format_stamp: "Formatted at 2026-09-28 08:42:33 on dist-test-slave-w6c0"
server_key: "0cd7f917cf1b3407957cc0894007cad1"
server_key_iv: "1bf594895de9bf867e913dd81e5aca89"
server_key_version: "encryptionkey@0"
I20260928 08:42:33.854905 21561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 08:42:33.858240 21576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:33.859660 21561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:42:33.859877 21561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "fd106f2a0a714f129b01112003e7cbbb"
format_stamp: "Formatted at 2026-09-28 08:42:33 on dist-test-slave-w6c0"
server_key: "0cd7f917cf1b3407957cc0894007cad1"
server_key_iv: "1bf594895de9bf867e913dd81e5aca89"
server_key_version: "encryptionkey@0"
I20260928 08:42:33.860098 21561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:33.877197 21561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:33.881721 21561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:33.882033 21561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:33.900300 21561 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41231
I20260928 08:42:33.900353 21627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41231 every 8 connection(s)
I20260928 08:42:33.901687 21561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:33.905460 21628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:33.906364 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 21561
I20260928 08:42:33.906565 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:33.907121 388 external_mini_cluster.cc:1428] Setting key 26fdd33de5311e2dbf56eaa36a2de0fb
I20260928 08:42:33.917979 21628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb: Bootstrap starting.
I20260928 08:42:33.920297 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:33.908966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:43293 (local address 127.0.0.1:41231)
0928 08:42:33.909726 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:33.909737 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:33.909775 (+ 38us) server_negotiation.cc:407] Connection header received
0928 08:42:33.911882 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:33.911906 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:33.912294 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:33.912684 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:33.913212 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:33.914453 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:33.915855 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:33.916526 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:33.916724 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:33.918011 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:33.918019 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:33.918032 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:33.918676 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:33.918753 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:33.918781 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:33.918953 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:33.919212 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 08:42:33.919386 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":317,"threads_started":1}
I20260928 08:42:33.921433 21628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:33.922509 21628 log.cc:824] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:33.924118 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:42:33.925212 21628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb: No bootstrap required, opened a new log
I20260928 08:42:33.933122 21628 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd106f2a0a714f129b01112003e7cbbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41231 } }
I20260928 08:42:33.933564 21628 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:33.933681 21628 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd106f2a0a714f129b01112003e7cbbb, State: Initialized, Role: FOLLOWER
I20260928 08:42:33.934257 21628 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd106f2a0a714f129b01112003e7cbbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41231 } }
I20260928 08:42:33.934551 21628 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:33.934715 21628 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:33.934903 21628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:33.934955 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:33.928511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:36417 (local address 127.0.0.1:41231)
0928 08:42:33.928690 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:33.928698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:33.929059 (+ 361us) server_negotiation.cc:407] Connection header received
0928 08:42:33.929257 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:33.929262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:33.929333 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:33.929459 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:33.929964 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:33.930847 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:33.932197 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:33.932632 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:33.932770 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:33.934001 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:33.934010 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:33.934016 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:33.934041 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:33.934119 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:33.934148 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:33.934282 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:33.934473 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 08:42:33.934572 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 08:42:33.937333 21628 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd106f2a0a714f129b01112003e7cbbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41231 } }
I20260928 08:42:33.937808 21628 leader_election.cc:302] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd106f2a0a714f129b01112003e7cbbb; no voters:
I20260928 08:42:33.938356 21628 leader_election.cc:288] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:33.938484 21638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:33.939241 21638 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb [term 1 LEADER]: Becoming Leader. State: Replica: fd106f2a0a714f129b01112003e7cbbb, State: Running, Role: LEADER
I20260928 08:42:33.939836 21638 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd106f2a0a714f129b01112003e7cbbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41231 } }
I20260928 08:42:33.940447 21628 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:33.942559 21640 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd106f2a0a714f129b01112003e7cbbb. Latest consensus state: current_term: 1 leader_uuid: "fd106f2a0a714f129b01112003e7cbbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd106f2a0a714f129b01112003e7cbbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41231 } } }
I20260928 08:42:33.942800 21639 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd106f2a0a714f129b01112003e7cbbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd106f2a0a714f129b01112003e7cbbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41231 } } }
I20260928 08:42:33.943066 21640 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:33.943120 21639 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:33.947052 21645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:33.951936 21645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:33.960778 21645 catalog_manager.cc:1347] Generated new cluster ID: 689b5732f85f41e6bef62e3f0261d98a
I20260928 08:42:33.960917 21645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:33.977993 21645 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:33.979401 21645 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:33.987169 21645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd106f2a0a714f129b01112003e7cbbb: Generated new TSK 0
I20260928 08:42:33.988101 21645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:34.005465 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 21561
W20260928 08:42:34.006666 21623 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.97:43293
W20260928 08:42:34.007429 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 21561: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:42:34.007568 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 21561
W20260928 08:42:34.025417 21546 connection.cc:535] client connection to 127.0.0.1:41231 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41231): Connection reset by peer (error 104)
2026-09-28T08:42:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1905 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:34 dist-test-slave-w6c0 krb5kdc[21659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:34 dist-test-slave-w6c0 krb5kdc[21659](info): set up 2 sockets
Sep 28 08:42:34 dist-test-slave-w6c0 krb5kdc[21659](info): commencing operation
krb5kdc: starting...
W20260928 08:42:35.131011 388 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:35 dist-test-slave-w6c0 krb5kdc[21659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:42:35.389040 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46481
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:35.717859 21672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:35.718160 21672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:35.718251 21672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:35.725625 21672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:35.725749 21672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:35.725797 21672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:35.725858 21672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:35.725901 21672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:35.736096 21672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:35.737347 21672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:35.738667 21672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:35.751572 21678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:35.751557 21677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:35.752054 21680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:35.752496 21672 server_base.cc:1034] running on GCE node
I20260928 08:42:35.753355 21672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:35.754923 21672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:35.756155 21672 hybrid_clock.cc:648] HybridClock initialized: now 1790584955756107 us; error 41 us; skew 500 ppm
I20260928 08:42:35.756569 21672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:35 dist-test-slave-w6c0 krb5kdc[21659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:35.759910 21672 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:42:35.761448 21672 webserver.cc:460] Webserver started at http://127.0.0.1:36905/ using document root <none> and password file <none>
I20260928 08:42:35.762240 21672 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:35.762372 21672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:35.762754 21672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:35.765823 21672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "56c5149659024bb4909351446d16bc5d"
format_stamp: "Formatted at 2026-09-28 08:42:35 on dist-test-slave-w6c0"
server_key: "192aa9f03a60b687ac9fd7e75d661836"
server_key_iv: "a59954a3e8550c27e05d13ee1f80d406"
server_key_version: "encryptionkey@0"
I20260928 08:42:35.766669 21672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "56c5149659024bb4909351446d16bc5d"
format_stamp: "Formatted at 2026-09-28 08:42:35 on dist-test-slave-w6c0"
server_key: "192aa9f03a60b687ac9fd7e75d661836"
server_key_iv: "a59954a3e8550c27e05d13ee1f80d406"
server_key_version: "encryptionkey@0"
I20260928 08:42:35.771884 21672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:42:35.775396 21687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:35.776759 21672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 08:42:35.776955 21672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "56c5149659024bb4909351446d16bc5d"
format_stamp: "Formatted at 2026-09-28 08:42:35 on dist-test-slave-w6c0"
server_key: "192aa9f03a60b687ac9fd7e75d661836"
server_key_iv: "a59954a3e8550c27e05d13ee1f80d406"
server_key_version: "encryptionkey@0"
I20260928 08:42:35.777159 21672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:35.783213 21672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:35.787674 21672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:35.787983 21672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:35.808224 21672 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46481
I20260928 08:42:35.808254 21738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46481 every 8 connection(s)
I20260928 08:42:35.809782 21672 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:35.813529 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 21672
I20260928 08:42:35.813731 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:35.813952 21739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:35.814276 388 external_mini_cluster.cc:1428] Setting key 330083da104a9cad86b5fdcd774c321c
I20260928 08:42:35.822217 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:35.816412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:36885 (local address 127.0.0.1:46481)
0928 08:42:35.817243 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:35.817254 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:35.817298 (+ 44us) server_negotiation.cc:407] Connection header received
0928 08:42:35.818930 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:35.818951 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:35.819280 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:35.819626 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:35.820712 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:35.820718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:35.820731 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:35.820745 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:35.820802 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:35.820824 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:35.820995 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:35.821099 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 08:42:35.821306 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":390,"threads_started":1}
I20260928 08:42:35.825822 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:42:35.827960 21739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d: Bootstrap starting.
I20260928 08:42:35.832037 21739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:35.833413 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:35.831111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:46513 (local address 127.0.0.1:46481)
0928 08:42:35.831259 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:35.831263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:35.831577 (+ 314us) server_negotiation.cc:407] Connection header received
0928 08:42:35.831762 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:35.831766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:35.831811 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:35.831881 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:35.832782 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:35.832786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:35.832789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:35.832803 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:35.832838 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:35.832852 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:35.832937 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:35.833061 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 08:42:35.833104 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 08:42:35.833403 21739 log.cc:824] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:35.836270 21739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d: No bootstrap required, opened a new log
I20260928 08:42:35.843382 21739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c5149659024bb4909351446d16bc5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260928 08:42:35.843744 21739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:35.843842 21739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c5149659024bb4909351446d16bc5d, State: Initialized, Role: FOLLOWER
I20260928 08:42:35.844411 21739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c5149659024bb4909351446d16bc5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260928 08:42:35.844637 21739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:35.844784 21739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:35.844944 21739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:35.847190 21739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c5149659024bb4909351446d16bc5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260928 08:42:35.847687 21739 leader_election.cc:302] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56c5149659024bb4909351446d16bc5d; no voters:
I20260928 08:42:35.848378 21739 leader_election.cc:288] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:35.848487 21749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:35.849251 21749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d [term 1 LEADER]: Becoming Leader. State: Replica: 56c5149659024bb4909351446d16bc5d, State: Running, Role: LEADER
I20260928 08:42:35.849783 21749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c5149659024bb4909351446d16bc5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260928 08:42:35.850585 21739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:35.852815 21750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56c5149659024bb4909351446d16bc5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c5149659024bb4909351446d16bc5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } } }
I20260928 08:42:35.852733 21751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56c5149659024bb4909351446d16bc5d. Latest consensus state: current_term: 1 leader_uuid: "56c5149659024bb4909351446d16bc5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c5149659024bb4909351446d16bc5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } } }
I20260928 08:42:35.853148 21751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:35.853148 21750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:35.855425 21756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:35.860414 21756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:35.871337 21756 catalog_manager.cc:1347] Generated new cluster ID: 7f5f136969b3467a8e0e594a71eae790
I20260928 08:42:35.871495 21756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:35.884904 21756 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:35.885334 21756 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:35.896440 21756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56c5149659024bb4909351446d16bc5d: Generated new TSK 0
I20260928 08:42:35.897295 21756 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:35.905208 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 21672
W20260928 08:42:35.906674 21734 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.97:36885
W20260928 08:42:35.907356 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 21672: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:42:35.907496 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 21672
2026-09-28T08:42:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1896 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:36 dist-test-slave-w6c0 krb5kdc[21770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:36 dist-test-slave-w6c0 krb5kdc[21770](info): set up 2 sockets
Sep 28 08:42:36 dist-test-slave-w6c0 krb5kdc[21770](info): commencing operation
krb5kdc: starting...
W20260928 08:42:37.011060 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:37 dist-test-slave-w6c0 krb5kdc[21770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:42:37.273839 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:37.601519 21783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:37.601788 21783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:37.601879 21783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:37.609042 21783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:37.609144 21783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:37.609191 21783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:37.609251 21783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:37.609295 21783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:37.619092 21783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42143
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:37.620265 21783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:37.621466 21783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:37.633035 21788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:37.633070 21789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:37.633553 21791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:37.634403 21783 server_base.cc:1034] running on GCE node
I20260928 08:42:37.635191 21783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:37.636583 21783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:37.637813 21783 hybrid_clock.cc:648] HybridClock initialized: now 1790584957637781 us; error 42 us; skew 500 ppm
I20260928 08:42:37.638161 21783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:37 dist-test-slave-w6c0 krb5kdc[21770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:37.641227 21783 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:42:37.642725 21783 webserver.cc:460] Webserver started at http://127.0.0.1:39703/ using document root <none> and password file <none>
I20260928 08:42:37.643394 21783 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:37.643536 21783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:37.643831 21783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:37.646745 21783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "c8fbbb5dd78145d78ad23ea60d36117b"
format_stamp: "Formatted at 2026-09-28 08:42:37 on dist-test-slave-w6c0"
server_key: "4e40b17485379c1ea80dd0820ac45498"
server_key_iv: "8639af9984138e4f2be7e4b5ff4ed7b7"
server_key_version: "encryptionkey@0"
I20260928 08:42:37.647490 21783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "c8fbbb5dd78145d78ad23ea60d36117b"
format_stamp: "Formatted at 2026-09-28 08:42:37 on dist-test-slave-w6c0"
server_key: "4e40b17485379c1ea80dd0820ac45498"
server_key_iv: "8639af9984138e4f2be7e4b5ff4ed7b7"
server_key_version: "encryptionkey@0"
I20260928 08:42:37.652294 21783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 08:42:37.655459 21798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:37.656591 21783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 08:42:37.656771 21783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "c8fbbb5dd78145d78ad23ea60d36117b"
format_stamp: "Formatted at 2026-09-28 08:42:37 on dist-test-slave-w6c0"
server_key: "4e40b17485379c1ea80dd0820ac45498"
server_key_iv: "8639af9984138e4f2be7e4b5ff4ed7b7"
server_key_version: "encryptionkey@0"
I20260928 08:42:37.656972 21783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:37.698372 21783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:37.702310 21783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:37.702670 21783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:37.720979 21783 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42143
I20260928 08:42:37.721021 21849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42143 every 8 connection(s)
I20260928 08:42:37.722338 21783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:37.725983 21850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:37.728279 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 21783
I20260928 08:42:37.728454 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:37.728972 388 external_mini_cluster.cc:1428] Setting key 646a9b5eaf1db6348227faa820ee7eb2
I20260928 08:42:37.738731 21850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b: Bootstrap starting.
I20260928 08:42:37.741951 21850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b: Neither blocks nor log segments found. Creating new log.
Sep 28 08:42:37 dist-test-slave-w6c0 krb5kdc[21770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 08:42:37.743048 21850 log.cc:824] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:37.746003 21850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b: No bootstrap required, opened a new log
I20260928 08:42:37.750476 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:37.730777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:59855 (local address 127.0.0.1:42143)
0928 08:42:37.731523 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:37.731534 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:37.731576 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:42:37.733173 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:37.733195 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:37.733587 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:37.733947 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:37.735387 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:37.736708 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:37.738016 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:37.738730 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:37.738928 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:37.743295 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:37.743303 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:37.743316 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:37.743358 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:37.746543 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:37.747294 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:37.747300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:37.747305 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:37.747488 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:37.747939 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:37.747945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:37.747947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:37.748427 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:37.748775 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:37.749248 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 08:42:37.749540 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":328,"threads_started":1}
I20260928 08:42:37.754328 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:42:37.754750 21850 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fbbb5dd78145d78ad23ea60d36117b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42143 } }
I20260928 08:42:37.755096 21850 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:37.755213 21850 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8fbbb5dd78145d78ad23ea60d36117b, State: Initialized, Role: FOLLOWER
I20260928 08:42:37.755738 21850 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fbbb5dd78145d78ad23ea60d36117b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42143 } }
I20260928 08:42:37.755975 21850 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:37.756125 21850 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:37.756305 21850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:37.758968 21850 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fbbb5dd78145d78ad23ea60d36117b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42143 } }
I20260928 08:42:37.759541 21850 leader_election.cc:302] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8fbbb5dd78145d78ad23ea60d36117b; no voters:
I20260928 08:42:37.760187 21850 leader_election.cc:288] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:37.760295 21860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:37.762172 21860 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b [term 1 LEADER]: Becoming Leader. State: Replica: c8fbbb5dd78145d78ad23ea60d36117b, State: Running, Role: LEADER
I20260928 08:42:37.762842 21850 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:37.762769 21860 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fbbb5dd78145d78ad23ea60d36117b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42143 } }
I20260928 08:42:37.768026 21862 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8fbbb5dd78145d78ad23ea60d36117b. Latest consensus state: current_term: 1 leader_uuid: "c8fbbb5dd78145d78ad23ea60d36117b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fbbb5dd78145d78ad23ea60d36117b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42143 } } }
I20260928 08:42:37.768517 21862 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:37.768298 21861 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8fbbb5dd78145d78ad23ea60d36117b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fbbb5dd78145d78ad23ea60d36117b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42143 } } }
I20260928 08:42:37.768771 21861 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:37.778563 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:37.758857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:47075 (local address 127.0.0.1:42143)
0928 08:42:37.759033 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:37.759040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:37.759482 (+ 442us) server_negotiation.cc:407] Connection header received
0928 08:42:37.759702 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:37.759710 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:37.759783 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:37.759891 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:37.761911 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:37.763162 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:37.764585 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:37.765028 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:37.765151 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:37.768194 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:37.768203 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:37.768207 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:37.768273 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:37.772988 (+ 4715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:37.774303 (+ 1315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:37.774312 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:37.774318 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:37.774539 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:37.775304 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:37.775313 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:37.775319 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:37.775855 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:37.776202 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:37.777796 (+ 1594us) server_negotiation.cc:299] Negotiation successful
0928 08:42:37.778042 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 08:42:37.783772 21867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:37.788745 21867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:37.796670 21867 catalog_manager.cc:1347] Generated new cluster ID: eb67c0b24b9b44cfa3039360fe42e2d9
I20260928 08:42:37.796783 21867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:37.817373 21867 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:37.818733 21867 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:37.829325 21867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8fbbb5dd78145d78ad23ea60d36117b: Generated new TSK 0
I20260928 08:42:37.830101 21867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:37.843529 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 21783
I20260928 08:42:37.844637 21845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:37.937825 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 21783
W20260928 08:42:37.955314 21768 connection.cc:535] client connection to 127.0.0.1:42143 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42143): Connection reset by peer (error 104)
2026-09-28T08:42:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2033 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:38 dist-test-slave-w6c0 krb5kdc[21882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:38 dist-test-slave-w6c0 krb5kdc[21882](info): set up 2 sockets
Sep 28 08:42:38 dist-test-slave-w6c0 krb5kdc[21882](info): commencing operation
krb5kdc: starting...
W20260928 08:42:39.035023 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:39 dist-test-slave-w6c0 krb5kdc[21882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:42:39.289762 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45671
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:39.627102 21895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:39.627424 21895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:39.627611 21895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:39.635020 21895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:39.635171 21895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:39.635267 21895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:39.635381 21895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:39.635469 21895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:39.645606 21895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45671
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:39.647064 21895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:39.648432 21895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:39.660681 21900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:39.660766 21903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:39.660727 21901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:39.661542 21895 server_base.cc:1034] running on GCE node
I20260928 08:42:39.662382 21895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:39.663900 21895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:39.665127 21895 hybrid_clock.cc:648] HybridClock initialized: now 1790584959665084 us; error 50 us; skew 500 ppm
I20260928 08:42:39.665518 21895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:39 dist-test-slave-w6c0 krb5kdc[21882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:39.672277 21895 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:42:39.673933 21895 webserver.cc:460] Webserver started at http://127.0.0.1:42397/ using document root <none> and password file <none>
I20260928 08:42:39.674724 21895 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:39.674834 21895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:39.675105 21895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:39.678121 21895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "1c55edc99e954082bfb32f7fe7e0dc19"
format_stamp: "Formatted at 2026-09-28 08:42:39 on dist-test-slave-w6c0"
server_key: "fa0bd6ca82dcf0a777314913773c94d2"
server_key_iv: "02ef9c58829c2710fb0160c6cadfa890"
server_key_version: "encryptionkey@0"
I20260928 08:42:39.678938 21895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "1c55edc99e954082bfb32f7fe7e0dc19"
format_stamp: "Formatted at 2026-09-28 08:42:39 on dist-test-slave-w6c0"
server_key: "fa0bd6ca82dcf0a777314913773c94d2"
server_key_iv: "02ef9c58829c2710fb0160c6cadfa890"
server_key_version: "encryptionkey@0"
I20260928 08:42:39.684267 21895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:42:39.687873 21910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:39.689350 21895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:42:39.689507 21895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "1c55edc99e954082bfb32f7fe7e0dc19"
format_stamp: "Formatted at 2026-09-28 08:42:39 on dist-test-slave-w6c0"
server_key: "fa0bd6ca82dcf0a777314913773c94d2"
server_key_iv: "02ef9c58829c2710fb0160c6cadfa890"
server_key_version: "encryptionkey@0"
I20260928 08:42:39.689687 21895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:39.703651 21895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:39.708571 21895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:39.708839 21895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:39.728036 21895 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45671
I20260928 08:42:39.728081 21961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45671 every 8 connection(s)
I20260928 08:42:39.729483 21895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:39.733222 21962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:39.736948 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 21895
I20260928 08:42:39.737124 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:39.737625 388 external_mini_cluster.cc:1428] Setting key d021fce0a8f6da8d5d1b63395d16bef8
I20260928 08:42:39.745193 21962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19: Bootstrap starting.
I20260928 08:42:39.748276 21962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:39.749357 21962 log.cc:824] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:39.750607 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:39.739472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:54613 (local address 127.0.0.1:45671)
0928 08:42:39.740203 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:39.740223 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:39.740264 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:42:39.741900 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:39.741922 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:39.742348 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:39.742790 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:39.743347 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:39.744605 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:39.745959 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:39.746722 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:39.746918 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:39.748176 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:39.748183 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:39.748197 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:39.748803 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:39.748873 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:39.748898 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:39.749060 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:39.749340 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 08:42:39.749529 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":316,"threads_started":1}
I20260928 08:42:39.752506 21962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19: No bootstrap required, opened a new log
I20260928 08:42:39.754571 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:42:39.761245 21962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c55edc99e954082bfb32f7fe7e0dc19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45671 } }
I20260928 08:42:39.761752 21962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:39.761857 21962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c55edc99e954082bfb32f7fe7e0dc19, State: Initialized, Role: FOLLOWER
I20260928 08:42:39.762576 21962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c55edc99e954082bfb32f7fe7e0dc19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45671 } }
I20260928 08:42:39.762867 21962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:39.763010 21962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:39.763260 21962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:39.765724 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:39.759077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:59783 (local address 127.0.0.1:45671)
0928 08:42:39.759304 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:39.759311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:39.759567 (+ 256us) server_negotiation.cc:407] Connection header received
0928 08:42:39.759745 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:39.759749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:39.759811 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:39.759893 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:39.760452 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:39.761203 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:39.763085 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:39.763620 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:39.763716 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:39.764957 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:39.764962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:39.764966 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:39.764987 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:39.765022 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:39.765038 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:39.765131 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:39.765329 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 08:42:39.765373 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 08:42:39.766381 21962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c55edc99e954082bfb32f7fe7e0dc19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45671 } }
I20260928 08:42:39.767004 21962 leader_election.cc:302] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c55edc99e954082bfb32f7fe7e0dc19; no voters:
I20260928 08:42:39.767627 21962 leader_election.cc:288] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:39.767763 21972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:39.768481 21972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19 [term 1 LEADER]: Becoming Leader. State: Replica: 1c55edc99e954082bfb32f7fe7e0dc19, State: Running, Role: LEADER
I20260928 08:42:39.769125 21972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c55edc99e954082bfb32f7fe7e0dc19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45671 } }
I20260928 08:42:39.769829 21962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:39.771955 21973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c55edc99e954082bfb32f7fe7e0dc19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c55edc99e954082bfb32f7fe7e0dc19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45671 } } }
I20260928 08:42:39.772276 21973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:39.773006 21974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c55edc99e954082bfb32f7fe7e0dc19. Latest consensus state: current_term: 1 leader_uuid: "1c55edc99e954082bfb32f7fe7e0dc19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c55edc99e954082bfb32f7fe7e0dc19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45671 } } }
I20260928 08:42:39.773313 21974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:39.774227 21977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:39.779816 21977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:39.789943 21977 catalog_manager.cc:1347] Generated new cluster ID: 13903638156f4a91aaeb6fd2c45ca7e9
I20260928 08:42:39.790071 21977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:39.810863 21977 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:39.812295 21977 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:39.820276 21977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c55edc99e954082bfb32f7fe7e0dc19: Generated new TSK 0
I20260928 08:42:39.821110 21977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:39.829576 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 21895
W20260928 08:42:39.830720 21957 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.97:54613
W20260928 08:42:39.831462 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 21895: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:42:39.831617 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 21895
W20260928 08:42:39.848896 21880 connection.cc:535] client connection to 127.0.0.1:45671 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45671): Connection reset by peer (error 104)
2026-09-28T08:42:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1893 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:39 dist-test-slave-w6c0 krb5kdc[21993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:39 dist-test-slave-w6c0 krb5kdc[21993](info): set up 2 sockets
Sep 28 08:42:39 dist-test-slave-w6c0 krb5kdc[21993](info): commencing operation
krb5kdc: starting...
W20260928 08:42:40.958976 388 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.005s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:41 dist-test-slave-w6c0 krb5kdc[21993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:42:41.194497 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39197
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:41.523898 22006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:41.524197 22006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:41.524396 22006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:41.531623 22006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:41.531740 22006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:41.531863 22006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:41.531986 22006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:41.532063 22006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:41.542008 22006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39197
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:41.543380 22006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:41.544701 22006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:41.556646 22014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:41.556675 22012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:41.556717 22011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:41.557374 22006 server_base.cc:1034] running on GCE node
I20260928 08:42:41.558147 22006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:41.559651 22006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:41.560884 22006 hybrid_clock.cc:648] HybridClock initialized: now 1790584961560827 us; error 47 us; skew 500 ppm
I20260928 08:42:41.561233 22006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:41 dist-test-slave-w6c0 krb5kdc[21993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:41.564301 22006 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:42:41.565713 22006 webserver.cc:460] Webserver started at http://127.0.0.1:41723/ using document root <none> and password file <none>
I20260928 08:42:41.566417 22006 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:41.566579 22006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:41.566854 22006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:41.569810 22006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "ffd35f87c46e42579e4001ed2aed69b6"
format_stamp: "Formatted at 2026-09-28 08:42:41 on dist-test-slave-w6c0"
server_key: "799f3dfc475acacab703ef761ffff925"
server_key_iv: "b79920c7f8ba5f76727ac8e8ac5d064a"
server_key_version: "encryptionkey@0"
I20260928 08:42:41.570655 22006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "ffd35f87c46e42579e4001ed2aed69b6"
format_stamp: "Formatted at 2026-09-28 08:42:41 on dist-test-slave-w6c0"
server_key: "799f3dfc475acacab703ef761ffff925"
server_key_iv: "b79920c7f8ba5f76727ac8e8ac5d064a"
server_key_version: "encryptionkey@0"
I20260928 08:42:41.575778 22006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:42:41.579298 22021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:41.580878 22006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:42:41.581027 22006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "ffd35f87c46e42579e4001ed2aed69b6"
format_stamp: "Formatted at 2026-09-28 08:42:41 on dist-test-slave-w6c0"
server_key: "799f3dfc475acacab703ef761ffff925"
server_key_iv: "b79920c7f8ba5f76727ac8e8ac5d064a"
server_key_version: "encryptionkey@0"
I20260928 08:42:41.581197 22006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:41.586884 22006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:41.591004 22006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:41.591250 22006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:41.610010 22006 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39197
I20260928 08:42:41.610100 22072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39197 every 8 connection(s)
I20260928 08:42:41.611544 22006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:41.615505 22073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:41.621466 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 22006
I20260928 08:42:41.621699 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:41.622138 388 external_mini_cluster.cc:1428] Setting key 53b517d66d70e0e09d29c55c35d5d30f
I20260928 08:42:41.626410 22073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6: Bootstrap starting.
I20260928 08:42:41.630084 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:41.624149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:56981 (local address 127.0.0.1:39197)
0928 08:42:41.624980 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:41.624991 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:41.625032 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:42:41.626804 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:41.626828 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:41.627152 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:41.627491 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:41.628534 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:41.628540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:41.628552 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:41.628565 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:41.628629 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:41.628648 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:41.628814 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:41.628969 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 08:42:41.629172 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":385,"threads_started":1}
I20260928 08:42:41.630163 22073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:41.631368 22073 log.cc:824] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:41.633803 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:42:41.634442 22073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6: No bootstrap required, opened a new log
I20260928 08:42:41.640632 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:41.638121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:40067 (local address 127.0.0.1:39197)
0928 08:42:41.638325 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:41.638332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:41.638769 (+ 437us) server_negotiation.cc:407] Connection header received
0928 08:42:41.638897 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:41.638902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:41.638966 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:41.639069 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:41.639984 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:41.639990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:41.639994 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:41.640011 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:41.640057 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:41.640081 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:41.640214 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:41.640267 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 08:42:41.640339 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:42:41.642719 22073 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd35f87c46e42579e4001ed2aed69b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39197 } }
I20260928 08:42:41.643071 22073 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:41.643137 22073 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd35f87c46e42579e4001ed2aed69b6, State: Initialized, Role: FOLLOWER
I20260928 08:42:41.643847 22073 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd35f87c46e42579e4001ed2aed69b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39197 } }
I20260928 08:42:41.644134 22073 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:41.644285 22073 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:41.644454 22073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:41.646675 22073 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd35f87c46e42579e4001ed2aed69b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39197 } }
I20260928 08:42:41.647171 22073 leader_election.cc:302] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffd35f87c46e42579e4001ed2aed69b6; no voters:
I20260928 08:42:41.647745 22073 leader_election.cc:288] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:41.647899 22083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:41.648573 22083 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6 [term 1 LEADER]: Becoming Leader. State: Replica: ffd35f87c46e42579e4001ed2aed69b6, State: Running, Role: LEADER
I20260928 08:42:41.649127 22083 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd35f87c46e42579e4001ed2aed69b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39197 } }
I20260928 08:42:41.649838 22073 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:41.651782 22084 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffd35f87c46e42579e4001ed2aed69b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd35f87c46e42579e4001ed2aed69b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39197 } } }
I20260928 08:42:41.652269 22084 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:41.652381 22085 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffd35f87c46e42579e4001ed2aed69b6. Latest consensus state: current_term: 1 leader_uuid: "ffd35f87c46e42579e4001ed2aed69b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd35f87c46e42579e4001ed2aed69b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39197 } } }
I20260928 08:42:41.652675 22085 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:41.654196 22088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:41.663993 22088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:41.673393 22088 catalog_manager.cc:1347] Generated new cluster ID: 31866a85b9914a3c8473fb06763136ce
I20260928 08:42:41.673588 22088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:41.701581 22088 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:41.702145 22088 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:41.711966 22088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffd35f87c46e42579e4001ed2aed69b6: Generated new TSK 0
I20260928 08:42:41.712842 22088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:41.720209 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 22006
W20260928 08:42:41.721753 22068 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.97:56981
W20260928 08:42:41.722442 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 22006: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:42:41.722609 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 22006
2026-09-28T08:42:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1891 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:41 dist-test-slave-w6c0 krb5kdc[22104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:41 dist-test-slave-w6c0 krb5kdc[22104](info): set up 2 sockets
Sep 28 08:42:41 dist-test-slave-w6c0 krb5kdc[22104](info): commencing operation
krb5kdc: starting...
W20260928 08:42:42.827081 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:42 dist-test-slave-w6c0 krb5kdc[22104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:42:43.074074 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40715
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:43.403069 22117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:43.403398 22117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:43.403558 22117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:43.410831 22117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:43.410934 22117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:43.411072 22117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:43.411185 22117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:43.411252 22117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:43.421064 22117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40715
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:43.422370 22117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:43.423640 22117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:43.435582 22122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:43.435582 22125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:43.435592 22123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:43.436532 22117 server_base.cc:1034] running on GCE node
I20260928 08:42:43.437320 22117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:43.438711 22117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:43.439908 22117 hybrid_clock.cc:648] HybridClock initialized: now 1790584963439868 us; error 41 us; skew 500 ppm
I20260928 08:42:43.440253 22117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:43 dist-test-slave-w6c0 krb5kdc[22104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:43.443727 22117 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:42:43.445541 22117 webserver.cc:460] Webserver started at http://127.0.97.62:35913/ using document root <none> and password file <none>
I20260928 08:42:43.446259 22117 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:43.446414 22117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:43.446755 22117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:43.449674 22117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "a44826fe8de34d249917fa948828e825"
format_stamp: "Formatted at 2026-09-28 08:42:43 on dist-test-slave-w6c0"
server_key: "d42f79d4420477a80cf09b9d1a707256"
server_key_iv: "ac4cf62d7e472da5e314472ca0aed585"
server_key_version: "encryptionkey@0"
I20260928 08:42:43.450444 22117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "a44826fe8de34d249917fa948828e825"
format_stamp: "Formatted at 2026-09-28 08:42:43 on dist-test-slave-w6c0"
server_key: "d42f79d4420477a80cf09b9d1a707256"
server_key_iv: "ac4cf62d7e472da5e314472ca0aed585"
server_key_version: "encryptionkey@0"
I20260928 08:42:43.455762 22117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:42:43.459306 22132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:43.460606 22117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 08:42:43.460785 22117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "a44826fe8de34d249917fa948828e825"
format_stamp: "Formatted at 2026-09-28 08:42:43 on dist-test-slave-w6c0"
server_key: "d42f79d4420477a80cf09b9d1a707256"
server_key_iv: "ac4cf62d7e472da5e314472ca0aed585"
server_key_version: "encryptionkey@0"
I20260928 08:42:43.460988 22117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:43.478893 22117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:43.482963 22117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:43.483300 22117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:43.502087 22117 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40715
I20260928 08:42:43.502115 22183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40715 every 8 connection(s)
I20260928 08:42:43.503558 22117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:43.507336 22184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:43.507622 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 22117
I20260928 08:42:43.507803 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:43.508245 388 external_mini_cluster.cc:1428] Setting key fe0553fe682e5d8226dab1b7305a587c
I20260928 08:42:43.521752 22184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825: Bootstrap starting.
Sep 28 08:42:43 dist-test-slave-w6c0 krb5kdc[22104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:42:43.525256 22184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:43.526362 22184 log.cc:824] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:43.530565 22184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825: No bootstrap required, opened a new log
I20260928 08:42:43.535638 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:43.510948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41755 (local address 127.0.97.62:40715)
0928 08:42:43.511782 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:43.511800 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:43.511867 (+ 67us) server_negotiation.cc:407] Connection header received
0928 08:42:43.514054 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:43.514083 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:43.514694 (+ 611us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:43.515195 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:43.516572 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:43.518605 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:43.520237 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:43.521280 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:43.521476 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:43.526366 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:43.526379 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:43.526402 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:43.526451 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:43.530713 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:43.531620 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:43.531628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:43.531636 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:43.531852 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:43.532350 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:43.532355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:43.532360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:43.533053 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:43.533414 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:43.533852 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 08:42:43.534251 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":325,"threads_started":1}
I20260928 08:42:43.539650 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:42:43.539995 22184 raft_consensus.cc:348] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44826fe8de34d249917fa948828e825" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40715 } }
I20260928 08:42:43.540337 22184 raft_consensus.cc:374] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:43.540446 22184 raft_consensus.cc:729] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44826fe8de34d249917fa948828e825, State: Initialized, Role: FOLLOWER
I20260928 08:42:43.540990 22184 consensus_queue.cc:260] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44826fe8de34d249917fa948828e825" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40715 } }
I20260928 08:42:43.541221 22184 raft_consensus.cc:388] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:43.541360 22184 raft_consensus.cc:482] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:43.541518 22184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:43.544368 22184 raft_consensus.cc:504] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44826fe8de34d249917fa948828e825" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40715 } }
I20260928 08:42:43.545001 22184 leader_election.cc:302] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a44826fe8de34d249917fa948828e825; no voters:
I20260928 08:42:43.545652 22184 leader_election.cc:288] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:43.545863 22194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:43.547611 22194 raft_consensus.cc:686] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825 [term 1 LEADER]: Becoming Leader. State: Replica: a44826fe8de34d249917fa948828e825, State: Running, Role: LEADER
I20260928 08:42:43.547857 22184 sys_catalog.cc:564] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:43.548244 22194 consensus_queue.cc:237] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44826fe8de34d249917fa948828e825" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40715 } }
I20260928 08:42:43.551054 22195 sys_catalog.cc:455] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a44826fe8de34d249917fa948828e825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44826fe8de34d249917fa948828e825" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40715 } } }
I20260928 08:42:43.551451 22195 sys_catalog.cc:458] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:43.556459 22196 sys_catalog.cc:455] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a44826fe8de34d249917fa948828e825. Latest consensus state: current_term: 1 leader_uuid: "a44826fe8de34d249917fa948828e825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44826fe8de34d249917fa948828e825" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40715 } } }
I20260928 08:42:43.557277 22196 sys_catalog.cc:458] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:43.564441 22200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:43.565593 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:43.544127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55617 (local address 127.0.97.62:40715)
0928 08:42:43.544293 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:43.544297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:43.544813 (+ 516us) server_negotiation.cc:407] Connection header received
0928 08:42:43.544947 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:43.544953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:43.545039 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:43.545155 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:43.547328 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:43.548575 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:43.550440 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:43.551293 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:43.551449 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:43.555827 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:43.555837 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:43.555844 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:43.555897 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:43.560792 (+ 4895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:43.562303 (+ 1511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:43.562315 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:43.562322 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:43.562605 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:43.563312 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:43.563320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:43.563326 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:43.563839 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:43.564217 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:43.564778 (+ 561us) server_negotiation.cc:299] Negotiation successful
0928 08:42:43.565194 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 08:42:43.569860 22200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:43.581836 22200 catalog_manager.cc:1347] Generated new cluster ID: 3c9d4bf1bfa04044837f35eb40c8ae2c
I20260928 08:42:43.582000 22200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:43.608364 22200 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:43.609874 22200 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:43.615806 22200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a44826fe8de34d249917fa948828e825: Generated new TSK 0
I20260928 08:42:43.616556 22200 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:43.628855 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 22117
I20260928 08:42:43.630064 22179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:43.724196 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 22117
W20260928 08:42:43.741976 22102 connection.cc:535] client connection to 127.0.97.62:40715 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:40715): Connection reset by peer (error 104)
2026-09-28T08:42:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2002 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:43 dist-test-slave-w6c0 krb5kdc[22216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:43 dist-test-slave-w6c0 krb5kdc[22216](info): set up 2 sockets
Sep 28 08:42:43 dist-test-slave-w6c0 krb5kdc[22216](info): commencing operation
krb5kdc: starting...
W20260928 08:42:44.847813 388 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:45 dist-test-slave-w6c0 krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:42:45.135094 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41641
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:45.503697 22229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:45.504071 22229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:45.504252 22229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:45.511868 22229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:45.512027 22229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:45.512140 22229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:45.512259 22229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:45.512337 22229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:45.522657 22229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41641
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:45.524243 22229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:45.525642 22229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:45.537869 22234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:45.537806 22235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:45.537936 22237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:45.538656 22229 server_base.cc:1034] running on GCE node
I20260928 08:42:45.539458 22229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:45.540887 22229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:45.542141 22229 hybrid_clock.cc:648] HybridClock initialized: now 1790584965542113 us; error 53 us; skew 500 ppm
I20260928 08:42:45.542573 22229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:45 dist-test-slave-w6c0 krb5kdc[22216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:45.545584 22229 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:42:45.547181 22229 webserver.cc:460] Webserver started at http://127.0.97.62:44297/ using document root <none> and password file <none>
I20260928 08:42:45.547874 22229 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:45.548018 22229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:45.548328 22229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:45.551455 22229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "fca4718a45ec40309c3e757a6f35a486"
format_stamp: "Formatted at 2026-09-28 08:42:45 on dist-test-slave-w6c0"
server_key: "a8d28f4988d75216686ef286b24c3bc7"
server_key_iv: "e324444ec38952645a89347aaaf694ee"
server_key_version: "encryptionkey@0"
I20260928 08:42:45.552299 22229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "fca4718a45ec40309c3e757a6f35a486"
format_stamp: "Formatted at 2026-09-28 08:42:45 on dist-test-slave-w6c0"
server_key: "a8d28f4988d75216686ef286b24c3bc7"
server_key_iv: "e324444ec38952645a89347aaaf694ee"
server_key_version: "encryptionkey@0"
I20260928 08:42:45.557454 22229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:42:45.561133 22244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:45.562490 22229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 08:42:45.562772 22229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "fca4718a45ec40309c3e757a6f35a486"
format_stamp: "Formatted at 2026-09-28 08:42:45 on dist-test-slave-w6c0"
server_key: "a8d28f4988d75216686ef286b24c3bc7"
server_key_iv: "e324444ec38952645a89347aaaf694ee"
server_key_version: "encryptionkey@0"
I20260928 08:42:45.562973 22229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:45.584282 22229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:45.588279 22229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:45.588608 22229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:45.607515 22229 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41641
I20260928 08:42:45.607547 22295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41641 every 8 connection(s)
I20260928 08:42:45.608865 22229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:45.612577 22296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:45.613400 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 22229
I20260928 08:42:45.613590 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:45.614040 388 external_mini_cluster.cc:1428] Setting key 82f8a563a2fd783c4244d8ac986611ed
I20260928 08:42:45.626132 22296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486: Bootstrap starting.
I20260928 08:42:45.627606 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:45.615763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37495 (local address 127.0.97.62:41641)
0928 08:42:45.616519 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:45.616530 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:45.616571 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:42:45.618469 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:45.618494 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:45.618948 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:45.619284 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:45.619887 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:45.621239 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:45.623150 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:45.623849 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:45.623985 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:45.625330 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:45.625338 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:45.625351 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:45.625889 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:45.625961 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:45.625986 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:45.626119 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:45.626453 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 08:42:45.626685 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":365,"threads_started":1}
I20260928 08:42:45.629601 22296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:45.630888 22296 log.cc:824] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:45.631489 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:42:45.633970 22296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486: No bootstrap required, opened a new log
I20260928 08:42:45.643288 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:45.636595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35459 (local address 127.0.97.62:41641)
0928 08:42:45.636750 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:45.636754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:45.637148 (+ 394us) server_negotiation.cc:407] Connection header received
0928 08:42:45.637324 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:45.637327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:45.637388 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:45.637468 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:45.638045 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:45.638863 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:45.640712 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:45.641154 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:45.641240 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:45.642429 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:45.642433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:45.642436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:45.642453 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:45.642567 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:45.642588 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:45.642677 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:45.642879 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 08:42:45.642956 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 08:42:45.644551 22296 raft_consensus.cc:348] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca4718a45ec40309c3e757a6f35a486" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41641 } }
I20260928 08:42:45.644970 22296 raft_consensus.cc:374] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:45.645092 22296 raft_consensus.cc:729] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca4718a45ec40309c3e757a6f35a486, State: Initialized, Role: FOLLOWER
I20260928 08:42:45.645658 22296 consensus_queue.cc:260] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca4718a45ec40309c3e757a6f35a486" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41641 } }
I20260928 08:42:45.645888 22296 raft_consensus.cc:388] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:45.646037 22296 raft_consensus.cc:482] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:45.646210 22296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:45.648425 22296 raft_consensus.cc:504] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca4718a45ec40309c3e757a6f35a486" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41641 } }
I20260928 08:42:45.648921 22296 leader_election.cc:302] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fca4718a45ec40309c3e757a6f35a486; no voters:
I20260928 08:42:45.649463 22296 leader_election.cc:288] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:45.649600 22306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:45.651532 22306 raft_consensus.cc:686] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486 [term 1 LEADER]: Becoming Leader. State: Replica: fca4718a45ec40309c3e757a6f35a486, State: Running, Role: LEADER
I20260928 08:42:45.652103 22306 consensus_queue.cc:237] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca4718a45ec40309c3e757a6f35a486" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41641 } }
I20260928 08:42:45.652405 22296 sys_catalog.cc:564] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:45.655215 22308 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fca4718a45ec40309c3e757a6f35a486. Latest consensus state: current_term: 1 leader_uuid: "fca4718a45ec40309c3e757a6f35a486" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca4718a45ec40309c3e757a6f35a486" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41641 } } }
I20260928 08:42:45.655750 22308 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:45.655889 22307 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fca4718a45ec40309c3e757a6f35a486" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca4718a45ec40309c3e757a6f35a486" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41641 } } }
I20260928 08:42:45.656224 22307 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:45.656612 22315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:45.662120 22315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:45.671020 22315 catalog_manager.cc:1347] Generated new cluster ID: 06449307ea6146b1b07268a9ea7af865
I20260928 08:42:45.671135 22315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:45.702692 22315 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:45.704068 22315 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:45.715160 22315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fca4718a45ec40309c3e757a6f35a486: Generated new TSK 0
I20260928 08:42:45.716033 22315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:45.723342 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 22229
W20260928 08:42:45.724571 22291 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37495
W20260928 08:42:45.725294 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 22229: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:42:45.725450 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 22229
W20260928 08:42:45.742763 22214 connection.cc:535] client connection to 127.0.97.62:41641 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:41641): Connection reset by peer (error 104)
2026-09-28T08:42:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2000 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:45 dist-test-slave-w6c0 krb5kdc[22327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:45 dist-test-slave-w6c0 krb5kdc[22327](info): set up 2 sockets
Sep 28 08:42:45 dist-test-slave-w6c0 krb5kdc[22327](info): commencing operation
krb5kdc: starting...
W20260928 08:42:46.815055 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:46 dist-test-slave-w6c0 krb5kdc[22327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:42:47.063868 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40545
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:47.393230 22340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:47.393523 22340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:47.393613 22340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:47.400882 22340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:47.401000 22340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:47.401046 22340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:47.401103 22340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:47.401145 22340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:47.410740 22340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40545
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:47.412017 22340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:47.413220 22340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:47.424784 22345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:47.424757 22346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:47.424911 22348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:47.425545 22340 server_base.cc:1034] running on GCE node
I20260928 08:42:47.426258 22340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:47.427685 22340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:47.428876 22340 hybrid_clock.cc:648] HybridClock initialized: now 1790584967428811 us; error 56 us; skew 500 ppm
I20260928 08:42:47.429181 22340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:47 dist-test-slave-w6c0 krb5kdc[22327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:47.432420 22340 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:42:47.433799 22340 webserver.cc:460] Webserver started at http://127.0.97.62:37199/ using document root <none> and password file <none>
I20260928 08:42:47.434443 22340 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:47.434617 22340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:47.434933 22340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:47.437876 22340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "2b50831375f6410e8d6b775bd1d73f53"
format_stamp: "Formatted at 2026-09-28 08:42:47 on dist-test-slave-w6c0"
server_key: "ab2f928ac61dd04797cc1d92bae1079f"
server_key_iv: "21c95c12e50cec3c528656b1f218cf50"
server_key_version: "encryptionkey@0"
I20260928 08:42:47.438721 22340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "2b50831375f6410e8d6b775bd1d73f53"
format_stamp: "Formatted at 2026-09-28 08:42:47 on dist-test-slave-w6c0"
server_key: "ab2f928ac61dd04797cc1d92bae1079f"
server_key_iv: "21c95c12e50cec3c528656b1f218cf50"
server_key_version: "encryptionkey@0"
I20260928 08:42:47.443821 22340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 08:42:47.447177 22355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:47.448419 22340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:42:47.448599 22340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "2b50831375f6410e8d6b775bd1d73f53"
format_stamp: "Formatted at 2026-09-28 08:42:47 on dist-test-slave-w6c0"
server_key: "ab2f928ac61dd04797cc1d92bae1079f"
server_key_iv: "21c95c12e50cec3c528656b1f218cf50"
server_key_version: "encryptionkey@0"
I20260928 08:42:47.448791 22340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:47.454180 22340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:47.458150 22340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:47.458443 22340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:47.476730 22340 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40545
I20260928 08:42:47.476775 22406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40545 every 8 connection(s)
I20260928 08:42:47.478168 22340 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:47.481776 22407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:47.485949 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 22340
I20260928 08:42:47.486123 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:47.486744 388 external_mini_cluster.cc:1428] Setting key 8105b8a0ec37fa6dbde637b890cb2db5
I20260928 08:42:47.494450 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:47.488504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51545 (local address 127.0.97.62:40545)
0928 08:42:47.489272 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:47.489282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:47.489323 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:42:47.490983 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:47.491006 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:47.491379 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:47.491713 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:47.492776 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:47.492782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:47.492794 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:47.492808 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:47.492869 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:47.492888 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:47.493129 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:47.493326 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 08:42:47.493528 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":364,"threads_started":1}
I20260928 08:42:47.495244 22407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53: Bootstrap starting.
I20260928 08:42:47.498165 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:42:47.498772 22407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:47.499758 22407 log.cc:824] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:47.503182 22407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53: No bootstrap required, opened a new log
I20260928 08:42:47.505151 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:47.502570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48337 (local address 127.0.97.62:40545)
0928 08:42:47.502775 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:47.502782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:47.503266 (+ 484us) server_negotiation.cc:407] Connection header received
0928 08:42:47.503433 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:47.503436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:47.503482 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:47.503555 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:47.504482 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:47.504486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:47.504489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:47.504503 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:47.504543 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:47.504557 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:47.504642 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:47.504783 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 08:42:47.504829 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 08:42:47.511600 22407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b50831375f6410e8d6b775bd1d73f53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40545 } }
I20260928 08:42:47.511993 22407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:47.512110 22407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b50831375f6410e8d6b775bd1d73f53, State: Initialized, Role: FOLLOWER
I20260928 08:42:47.512696 22407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b50831375f6410e8d6b775bd1d73f53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40545 } }
I20260928 08:42:47.512899 22407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:47.513046 22407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:47.513244 22407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:47.515637 22407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b50831375f6410e8d6b775bd1d73f53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40545 } }
I20260928 08:42:47.516152 22407 leader_election.cc:302] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b50831375f6410e8d6b775bd1d73f53; no voters:
I20260928 08:42:47.516706 22407 leader_election.cc:288] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:47.516855 22417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:47.517517 22417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53 [term 1 LEADER]: Becoming Leader. State: Replica: 2b50831375f6410e8d6b775bd1d73f53, State: Running, Role: LEADER
I20260928 08:42:47.518059 22417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b50831375f6410e8d6b775bd1d73f53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40545 } }
I20260928 08:42:47.518858 22407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:47.520936 22418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b50831375f6410e8d6b775bd1d73f53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b50831375f6410e8d6b775bd1d73f53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40545 } } }
I20260928 08:42:47.521020 22419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b50831375f6410e8d6b775bd1d73f53. Latest consensus state: current_term: 1 leader_uuid: "2b50831375f6410e8d6b775bd1d73f53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b50831375f6410e8d6b775bd1d73f53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40545 } } }
I20260928 08:42:47.521341 22418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:47.521430 22419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:47.524621 22423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:47.529633 22423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:47.538925 22423 catalog_manager.cc:1347] Generated new cluster ID: 5e0ac1ddcc194b0b96226129f6cc5a7e
I20260928 08:42:47.539047 22423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:47.558372 22423 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:47.558810 22423 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:47.574541 22423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b50831375f6410e8d6b775bd1d73f53: Generated new TSK 0
I20260928 08:42:47.575690 22423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:47.590337 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 22340
W20260928 08:42:47.591604 22402 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51545
W20260928 08:42:47.592330 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 22340: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:42:47.592497 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 22340
2026-09-28T08:42:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1866 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:47 dist-test-slave-w6c0 krb5kdc[22438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:47 dist-test-slave-w6c0 krb5kdc[22438](info): set up 2 sockets
Sep 28 08:42:47 dist-test-slave-w6c0 krb5kdc[22438](info): commencing operation
krb5kdc: starting...
W20260928 08:42:48.695079 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:48 dist-test-slave-w6c0 krb5kdc[22438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:42:48.949715 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40845
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:49.277848 22451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:49.278141 22451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:49.278318 22451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:49.285616 22451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:49.285730 22451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:49.285837 22451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:49.285941 22451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:49.286015 22451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:49.296159 22451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:49.297508 22451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:49.298970 22451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:49.310555 22456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:49.310603 22459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:49.310627 22457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:49.311641 22451 server_base.cc:1034] running on GCE node
I20260928 08:42:49.312448 22451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:49.313858 22451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:49.315079 22451 hybrid_clock.cc:648] HybridClock initialized: now 1790584969315041 us; error 53 us; skew 500 ppm
I20260928 08:42:49.315434 22451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:49 dist-test-slave-w6c0 krb5kdc[22438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:49.318431 22451 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:42:49.319886 22451 webserver.cc:460] Webserver started at http://127.0.97.62:40875/ using document root <none> and password file <none>
I20260928 08:42:49.320531 22451 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:49.320629 22451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:49.320874 22451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:49.323782 22451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "171da82fb6544284911918c82bdb26a9"
format_stamp: "Formatted at 2026-09-28 08:42:49 on dist-test-slave-w6c0"
server_key: "682066d425af62399ed8aa0c5cc42605"
server_key_iv: "c559e65e719dc42ecf5435a9f478fcea"
server_key_version: "encryptionkey@0"
I20260928 08:42:49.324486 22451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "171da82fb6544284911918c82bdb26a9"
format_stamp: "Formatted at 2026-09-28 08:42:49 on dist-test-slave-w6c0"
server_key: "682066d425af62399ed8aa0c5cc42605"
server_key_iv: "c559e65e719dc42ecf5435a9f478fcea"
server_key_version: "encryptionkey@0"
I20260928 08:42:49.329507 22451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 08:42:49.332926 22466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:49.334420 22451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:42:49.334743 22451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "171da82fb6544284911918c82bdb26a9"
format_stamp: "Formatted at 2026-09-28 08:42:49 on dist-test-slave-w6c0"
server_key: "682066d425af62399ed8aa0c5cc42605"
server_key_iv: "c559e65e719dc42ecf5435a9f478fcea"
server_key_version: "encryptionkey@0"
I20260928 08:42:49.334928 22451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:49.365540 22451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:49.369614 22451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:49.369913 22451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:49.388938 22451 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40845
I20260928 08:42:49.388958 22517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40845 every 8 connection(s)
I20260928 08:42:49.390338 22451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:49.394093 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 22451
I20260928 08:42:49.394292 22518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:49.394315 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:49.394865 388 external_mini_cluster.cc:1428] Setting key 420a4cfe0f854813b4f2802676ee0c2f
I20260928 08:42:49.409323 22518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9: Bootstrap starting.
Sep 28 08:42:49 dist-test-slave-w6c0 krb5kdc[22438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:42:49.412777 22518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:49.413879 22518 log.cc:824] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:49.416736 22518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9: No bootstrap required, opened a new log
I20260928 08:42:49.418210 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:49.396608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55961 (local address 127.0.97.62:40845)
0928 08:42:49.397487 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:49.397499 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:49.397540 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:42:49.399362 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:49.399383 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:49.399792 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:49.400128 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:49.401898 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:49.403275 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:49.405132 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:49.405842 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:49.405984 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:49.410948 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:49.410957 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:49.410974 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:49.411014 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:49.414239 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:49.415017 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:49.415023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:49.415029 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:49.415193 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:49.415567 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:49.415572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:49.415576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:49.416095 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:49.416413 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:49.416845 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 08:42:49.417276 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":421,"threads_started":1}
I20260928 08:42:49.421967 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:42:49.425026 22518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171da82fb6544284911918c82bdb26a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40845 } }
I20260928 08:42:49.425396 22518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:49.425524 22518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171da82fb6544284911918c82bdb26a9, State: Initialized, Role: FOLLOWER
I20260928 08:42:49.426189 22518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171da82fb6544284911918c82bdb26a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40845 } }
I20260928 08:42:49.426471 22518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:49.426676 22518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:49.426880 22518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:49.430006 22518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171da82fb6544284911918c82bdb26a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40845 } }
I20260928 08:42:49.430660 22518 leader_election.cc:302] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 171da82fb6544284911918c82bdb26a9; no voters:
I20260928 08:42:49.431316 22518 leader_election.cc:288] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:49.431445 22528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:49.432253 22528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9 [term 1 LEADER]: Becoming Leader. State: Replica: 171da82fb6544284911918c82bdb26a9, State: Running, Role: LEADER
I20260928 08:42:49.432825 22528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171da82fb6544284911918c82bdb26a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40845 } }
I20260928 08:42:49.433883 22518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:49.436504 22530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "171da82fb6544284911918c82bdb26a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171da82fb6544284911918c82bdb26a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40845 } } }
I20260928 08:42:49.436899 22530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:49.437119 22529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 171da82fb6544284911918c82bdb26a9. Latest consensus state: current_term: 1 leader_uuid: "171da82fb6544284911918c82bdb26a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171da82fb6544284911918c82bdb26a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40845 } } }
I20260928 08:42:49.437420 22529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:49.444768 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:49.426372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48067 (local address 127.0.97.62:40845)
0928 08:42:49.426640 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:49.426649 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:49.427103 (+ 454us) server_negotiation.cc:407] Connection header received
0928 08:42:49.427283 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:49.427286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:49.427350 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:49.427422 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:49.429206 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:49.429906 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:49.431822 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:49.432561 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:49.432685 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:49.435568 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:49.435582 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:49.435586 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:49.435623 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:49.440593 (+ 4970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:49.441801 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:49.441809 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:49.441831 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:49.442018 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:49.442535 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:49.442540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:49.442543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:49.442999 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:49.443331 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:49.443708 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 08:42:49.444025 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260928 08:42:49.447214 22543 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:42:49.447359 22543 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:42:49.452152 22536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:49.454339 22536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:49.463411 22536 catalog_manager.cc:1347] Generated new cluster ID: eebffde5383e4214955acf2c7bb8e04d
I20260928 08:42:49.463533 22536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:49.481402 22536 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:49.483227 22536 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:49.491462 22536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 171da82fb6544284911918c82bdb26a9: Generated new TSK 0
I20260928 08:42:49.492511 22536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:49.508260 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 22451
I20260928 08:42:49.509311 22513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:49.604985 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 22451
2026-09-28T08:42:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2013 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:49 dist-test-slave-w6c0 krb5kdc[22550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:49 dist-test-slave-w6c0 krb5kdc[22550](info): set up 2 sockets
Sep 28 08:42:49 dist-test-slave-w6c0 krb5kdc[22550](info): commencing operation
krb5kdc: starting...
W20260928 08:42:50.711018 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:50 dist-test-slave-w6c0 krb5kdc[22550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:42:50.956058 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38765
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:51.283411 22563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:51.283687 22563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:51.283876 22563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:51.291006 22563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:51.291110 22563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:51.291214 22563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:51.291317 22563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:51.291400 22563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:51.300962 22563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:51.302207 22563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:51.303567 22563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:51.315076 22568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:51.315017 22569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:51.314992 22571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:51.316017 22563 server_base.cc:1034] running on GCE node
I20260928 08:42:51.316758 22563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:51.318070 22563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:51.319322 22563 hybrid_clock.cc:648] HybridClock initialized: now 1790584971319285 us; error 51 us; skew 500 ppm
I20260928 08:42:51.319660 22563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:51 dist-test-slave-w6c0 krb5kdc[22550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:51.331888 22563 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:42:51.333464 22563 webserver.cc:460] Webserver started at http://127.0.97.62:32771/ using document root <none> and password file <none>
I20260928 08:42:51.334128 22563 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:51.334227 22563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:51.334473 22563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:51.337450 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "c4d981e2528d48efa24e33b6c7bf7402"
format_stamp: "Formatted at 2026-09-28 08:42:51 on dist-test-slave-w6c0"
server_key: "892dd709df3282e834fc5c171de071b1"
server_key_iv: "d80cd13c5cb794ab18d554e67924a75f"
server_key_version: "encryptionkey@0"
I20260928 08:42:51.338143 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "c4d981e2528d48efa24e33b6c7bf7402"
format_stamp: "Formatted at 2026-09-28 08:42:51 on dist-test-slave-w6c0"
server_key: "892dd709df3282e834fc5c171de071b1"
server_key_iv: "d80cd13c5cb794ab18d554e67924a75f"
server_key_version: "encryptionkey@0"
I20260928 08:42:51.343353 22563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260928 08:42:51.346578 22578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:51.347802 22563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:42:51.347934 22563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "c4d981e2528d48efa24e33b6c7bf7402"
format_stamp: "Formatted at 2026-09-28 08:42:51 on dist-test-slave-w6c0"
server_key: "892dd709df3282e834fc5c171de071b1"
server_key_iv: "d80cd13c5cb794ab18d554e67924a75f"
server_key_version: "encryptionkey@0"
I20260928 08:42:51.348093 22563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:51.397928 22563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:51.401858 22563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:51.402099 22563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:51.420539 22563 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38765
I20260928 08:42:51.420585 22629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38765 every 8 connection(s)
I20260928 08:42:51.421907 22563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:51.425598 22630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:51.430244 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 22563
I20260928 08:42:51.430426 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:51.430960 388 external_mini_cluster.cc:1428] Setting key a307fd23f518a8c21ed6763d37ca5b9b
I20260928 08:42:51.436584 22630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402: Bootstrap starting.
I20260928 08:42:51.440049 22630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:51.441298 22630 log.cc:824] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:51.444044 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:51.432651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59357 (local address 127.0.97.62:38765)
0928 08:42:51.433387 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:51.433399 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:51.433451 (+ 52us) server_negotiation.cc:407] Connection header received
0928 08:42:51.435196 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:51.435216 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:51.435588 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:51.435874 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:51.436434 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:51.437763 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:51.439663 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:51.440381 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:51.440511 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:51.441692 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:51.441698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:51.441710 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:51.442474 (+ 764us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:51.442599 (+ 125us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:51.442638 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:51.442746 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:51.442944 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 08:42:51.443185 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":341,"threads_started":1}
I20260928 08:42:51.444291 22630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402: No bootstrap required, opened a new log
I20260928 08:42:51.447499 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:42:51.454180 22630 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d981e2528d48efa24e33b6c7bf7402" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38765 } }
I20260928 08:42:51.454552 22630 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:51.454695 22630 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d981e2528d48efa24e33b6c7bf7402, State: Initialized, Role: FOLLOWER
I20260928 08:42:51.455281 22630 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d981e2528d48efa24e33b6c7bf7402" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38765 } }
I20260928 08:42:51.455535 22630 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:51.455689 22630 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:51.455868 22630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:51.457973 22630 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d981e2528d48efa24e33b6c7bf7402" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38765 } }
I20260928 08:42:51.458453 22630 leader_election.cc:302] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4d981e2528d48efa24e33b6c7bf7402; no voters:
I20260928 08:42:51.458750 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:51.452535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49827 (local address 127.0.97.62:38765)
0928 08:42:51.452758 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:51.452766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:51.453164 (+ 398us) server_negotiation.cc:407] Connection header received
0928 08:42:51.453342 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:51.453346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:51.453401 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:51.453500 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:51.454104 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:51.454946 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:51.456347 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:51.456851 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:51.456993 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:51.457991 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:51.457997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:51.458001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:51.458023 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:51.458061 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:51.458075 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:51.458163 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:51.458294 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 08:42:51.458369 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 08:42:51.459386 22630 leader_election.cc:288] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:51.459564 22640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:51.460637 22640 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402 [term 1 LEADER]: Becoming Leader. State: Replica: c4d981e2528d48efa24e33b6c7bf7402, State: Running, Role: LEADER
I20260928 08:42:51.461095 22640 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d981e2528d48efa24e33b6c7bf7402" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38765 } }
I20260928 08:42:51.461364 22630 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:51.463374 22642 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4d981e2528d48efa24e33b6c7bf7402. Latest consensus state: current_term: 1 leader_uuid: "c4d981e2528d48efa24e33b6c7bf7402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d981e2528d48efa24e33b6c7bf7402" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38765 } } }
I20260928 08:42:51.463687 22642 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:51.464094 22641 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4d981e2528d48efa24e33b6c7bf7402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d981e2528d48efa24e33b6c7bf7402" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38765 } } }
I20260928 08:42:51.464411 22641 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:51.466815 22647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:51.471688 22647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:51.479868 22647 catalog_manager.cc:1347] Generated new cluster ID: fb66d65d094f451daf46d6de92374ea1
I20260928 08:42:51.479971 22647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:51.514389 22647 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:51.516204 22647 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:51.525743 22647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4d981e2528d48efa24e33b6c7bf7402: Generated new TSK 0
I20260928 08:42:51.526728 22647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:51.534127 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 22563
W20260928 08:42:51.535187 22625 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59357
W20260928 08:42:51.535908 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 22563: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:42:51.536053 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 22563
2026-09-28T08:42:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1926 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:51 dist-test-slave-w6c0 krb5kdc[22661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:51 dist-test-slave-w6c0 krb5kdc[22661](info): set up 2 sockets
Sep 28 08:42:51 dist-test-slave-w6c0 krb5kdc[22661](info): commencing operation
krb5kdc: starting...
W20260928 08:42:52.630980 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:52 dist-test-slave-w6c0 krb5kdc[22661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:42:52.883594 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45305
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:53.216863 22674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:53.217165 22674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:53.217258 22674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:53.224617 22674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:53.224742 22674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:53.224790 22674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:53.224861 22674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:53.224907 22674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:53.234861 22674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:53.236320 22674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:53.237720 22674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:53.249931 22680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:53.249995 22679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:53.250023 22682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:53.250890 22674 server_base.cc:1034] running on GCE node
I20260928 08:42:53.251703 22674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:53.253182 22674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:53.254467 22674 hybrid_clock.cc:648] HybridClock initialized: now 1790584973254393 us; error 60 us; skew 500 ppm
I20260928 08:42:53.254918 22674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:53 dist-test-slave-w6c0 krb5kdc[22661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:53.258111 22674 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:42:53.259708 22674 webserver.cc:460] Webserver started at http://127.0.97.62:41639/ using document root <none> and password file <none>
I20260928 08:42:53.260411 22674 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:53.260567 22674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:53.260882 22674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:53.263988 22674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "3b6606f136a34cd2878926b2316f7d8c"
format_stamp: "Formatted at 2026-09-28 08:42:53 on dist-test-slave-w6c0"
server_key: "a5051b87e5e8bf18bb5caca95dae135c"
server_key_iv: "aae21ff9b2949026167e21469c3d85d8"
server_key_version: "encryptionkey@0"
I20260928 08:42:53.264766 22674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "3b6606f136a34cd2878926b2316f7d8c"
format_stamp: "Formatted at 2026-09-28 08:42:53 on dist-test-slave-w6c0"
server_key: "a5051b87e5e8bf18bb5caca95dae135c"
server_key_iv: "aae21ff9b2949026167e21469c3d85d8"
server_key_version: "encryptionkey@0"
I20260928 08:42:53.269917 22674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260928 08:42:53.273516 22689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:53.274881 22674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:42:53.275076 22674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "3b6606f136a34cd2878926b2316f7d8c"
format_stamp: "Formatted at 2026-09-28 08:42:53 on dist-test-slave-w6c0"
server_key: "a5051b87e5e8bf18bb5caca95dae135c"
server_key_iv: "aae21ff9b2949026167e21469c3d85d8"
server_key_version: "encryptionkey@0"
I20260928 08:42:53.275290 22674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:53.280907 22674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:53.285312 22674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:53.285640 22674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:53.304411 22674 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45305
I20260928 08:42:53.304503 22740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45305 every 8 connection(s)
I20260928 08:42:53.305825 22674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:53.308080 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 22674
I20260928 08:42:53.308247 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:53.308772 388 external_mini_cluster.cc:1428] Setting key 8f2f31adcfc295329176868377843976
I20260928 08:42:53.310269 22741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:53.316226 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:53.310604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49483 (local address 127.0.97.62:45305)
0928 08:42:53.311335 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:53.311346 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:53.311383 (+ 37us) server_negotiation.cc:407] Connection header received
0928 08:42:53.312910 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:53.312929 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:53.313248 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:53.313656 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:53.314697 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:53.314702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:53.314718 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:53.314737 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:53.314807 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:53.314826 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:53.315062 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:53.315199 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 08:42:53.315368 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":324,"threads_started":1}
I20260928 08:42:53.319978 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:42:53.322618 22741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c: Bootstrap starting.
I20260928 08:42:53.326237 22741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:53.326908 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:53.324275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55643 (local address 127.0.97.62:45305)
0928 08:42:53.324472 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:53.324478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:53.324773 (+ 295us) server_negotiation.cc:407] Connection header received
0928 08:42:53.324937 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:53.324940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:53.324986 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:53.325058 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:53.326086 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:53.326090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:53.326093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:42:53.326109 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:53.326153 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:42:53.326168 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:53.326267 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:53.326494 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 08:42:53.326621 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:42:53.327577 22741 log.cc:824] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:53.330305 22741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c: No bootstrap required, opened a new log
I20260928 08:42:53.337893 22741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6606f136a34cd2878926b2316f7d8c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45305 } }
I20260928 08:42:53.338398 22741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:42:53.338775 22741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6606f136a34cd2878926b2316f7d8c, State: Initialized, Role: FOLLOWER
I20260928 08:42:53.339458 22741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6606f136a34cd2878926b2316f7d8c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45305 } }
I20260928 08:42:53.339687 22741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:42:53.339963 22741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:42:53.340183 22741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:53.342358 22741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6606f136a34cd2878926b2316f7d8c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45305 } }
I20260928 08:42:53.342917 22741 leader_election.cc:302] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b6606f136a34cd2878926b2316f7d8c; no voters:
I20260928 08:42:53.343611 22741 leader_election.cc:288] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:42:53.343673 22751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:53.344846 22751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c [term 1 LEADER]: Becoming Leader. State: Replica: 3b6606f136a34cd2878926b2316f7d8c, State: Running, Role: LEADER
I20260928 08:42:53.345468 22751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6606f136a34cd2878926b2316f7d8c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45305 } }
I20260928 08:42:53.345832 22741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:53.348443 22753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b6606f136a34cd2878926b2316f7d8c. Latest consensus state: current_term: 1 leader_uuid: "3b6606f136a34cd2878926b2316f7d8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6606f136a34cd2878926b2316f7d8c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45305 } } }
I20260928 08:42:53.348868 22753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:53.348400 22752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b6606f136a34cd2878926b2316f7d8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6606f136a34cd2878926b2316f7d8c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45305 } } }
I20260928 08:42:53.349123 22752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:53.352603 22759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:53.358006 22759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:53.366593 22759 catalog_manager.cc:1347] Generated new cluster ID: a284f123fabd49a4bc43b71da6c9f88d
I20260928 08:42:53.366715 22759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:53.398553 22759 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:53.398999 22759 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:53.413298 22759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b6606f136a34cd2878926b2316f7d8c: Generated new TSK 0
I20260928 08:42:53.414252 22759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:42:53.419929 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 22674
W20260928 08:42:53.421775 22736 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49483
W20260928 08:42:53.422876 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 22674: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:42:53.423032 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 22674
2026-09-28T08:42:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1888 ms)
[----------] 18 tests from AuthTokenIssuingTest (35018 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:53 dist-test-slave-w6c0 krb5kdc[22772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:53 dist-test-slave-w6c0 krb5kdc[22772](info): set up 2 sockets
Sep 28 08:42:53 dist-test-slave-w6c0 krb5kdc[22772](info): commencing operation
krb5kdc: starting...
W20260928 08:42:54.527151 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:54 dist-test-slave-w6c0 krb5kdc[22772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:42:54.780318 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34695
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34695,127.0.97.61:44819,127.0.97.60:42125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:42:55.120740 22785 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:42:55.121098 22785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:55.121232 22785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:55.121385 22785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:55.128762 22785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:55.128861 22785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:55.128952 22785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:55.129033 22785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:55.129143 22785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:55.139137 22785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34695,127.0.97.61:44819,127.0.97.60:42125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:55.140439 22785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:55.141719 22785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:55.153545 22790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:55.153575 22791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:55.153611 22793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:55.154218 22785 server_base.cc:1034] running on GCE node
I20260928 08:42:55.155048 22785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:55.156546 22785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:55.157796 22785 hybrid_clock.cc:648] HybridClock initialized: now 1790584975157726 us; error 42 us; skew 500 ppm
I20260928 08:42:55.158121 22785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:55 dist-test-slave-w6c0 krb5kdc[22772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:55.161314 22785 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:42:55.162861 22785 webserver.cc:460] Webserver started at http://127.0.97.62:37751/ using document root <none> and password file <none>
I20260928 08:42:55.163558 22785 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:55.163692 22785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:55.164006 22785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:55.167016 22785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "07bd497d0c7044888fecd45b6cb77296"
format_stamp: "Formatted at 2026-09-28 08:42:55 on dist-test-slave-w6c0"
server_key: "7fc35889c3635f2ba44f98440b839b19"
server_key_iv: "9ef3854015bcae5be605fb48f41bfd51"
server_key_version: "encryptionkey@0"
I20260928 08:42:55.167802 22785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "07bd497d0c7044888fecd45b6cb77296"
format_stamp: "Formatted at 2026-09-28 08:42:55 on dist-test-slave-w6c0"
server_key: "7fc35889c3635f2ba44f98440b839b19"
server_key_iv: "9ef3854015bcae5be605fb48f41bfd51"
server_key_version: "encryptionkey@0"
I20260928 08:42:55.172899 22785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:42:55.176412 22800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:55.177831 22785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:42:55.178032 22785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "07bd497d0c7044888fecd45b6cb77296"
format_stamp: "Formatted at 2026-09-28 08:42:55 on dist-test-slave-w6c0"
server_key: "7fc35889c3635f2ba44f98440b839b19"
server_key_iv: "9ef3854015bcae5be605fb48f41bfd51"
server_key_version: "encryptionkey@0"
I20260928 08:42:55.178293 22785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:55.192060 22785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:55.197247 22785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:55.197564 22785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:55.218900 22785 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34695
I20260928 08:42:55.218923 22851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34695 every 8 connection(s)
I20260928 08:42:55.220233 22785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:42:55.223917 22852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:55.225919 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 22785
I20260928 08:42:55.226143 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:42:55.226979 388 external_mini_cluster.cc:1428] Setting key 55e972a3e94975018e65b26e21a9b133
I20260928 08:42:55.233239 22852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:42:55 dist-test-slave-w6c0 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260928 08:42:55.261786 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.237139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34695 (local address 127.0.0.1:48308)
0928 08:42:55.243426 (+ 6287us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.243468 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:55.245255 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:55.246077 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:55.246084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:55.246833 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:55.247166 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:55.247183 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:55.248767 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:55.248773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:55.249829 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:55.249841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:55.250830 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:55.250837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:55.251007 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:55.251754 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:42:55.251777 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:55.254485 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:55.257968 (+ 3483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:55.257978 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:55.257987 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:55.258379 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:55.258817 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:55.258820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:55.258822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:55.258974 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:55.259939 (+ 965us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:55.259945 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:55.260147 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 08:42:55.260363 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 08:42:55.260751 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6069,"thread_start_us":5877,"threads_started":1}
I20260928 08:42:55.261826 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.237139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.0.97.62:34695)
0928 08:42:55.242994 (+ 5855us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:55.243006 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:55.243601 (+ 595us) server_negotiation.cc:407] Connection header received
0928 08:42:55.245622 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:55.245630 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:55.245852 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:55.246053 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:55.247400 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:55.248602 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:55.249973 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:55.250591 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:55.250791 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:55.254762 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:55.254770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:55.254784 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:55.254814 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:55.257847 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:55.258552 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:55.258562 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:55.258566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:55.258718 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:55.259135 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:55.259140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:55.259142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:55.259788 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:55.260004 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:55.260558 (+ 554us) server_negotiation.cc:299] Negotiation successful
0928 08:42:55.260957 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":5661,"spinlock_wait_cycles":15744,"thread_start_us":5453,"threads_started":1}
I20260928 08:42:55.264714 22852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:42:55.265755 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.265364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44819 (local address 127.0.0.1:38874)
0928 08:42:55.265548 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.265652 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 08:42:55.265990 22802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44819: connect: Connection refused (error 111)
I20260928 08:42:55.267490 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.267157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44819 (local address 127.0.0.1:38882)
0928 08:42:55.267314 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.267374 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 08:42:55.267704 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44819: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44819: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260928 08:42:55.290143 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44819
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34695,127.0.97.61:44819,127.0.97.60:42125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:42:55.317065 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } attempt: 1
I20260928 08:42:55.318436 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.317864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44819 (local address 127.0.0.1:38898)
0928 08:42:55.318101 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.318226 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 08:42:55.319581 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.319189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44819 (local address 127.0.0.1:38908)
0928 08:42:55.319348 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.319430 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 08:42:55.319811 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44819: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44819: connect: Connection refused (error 111)
I20260928 08:42:55.388204 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } attempt: 2
I20260928 08:42:55.389544 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.388942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44819 (local address 127.0.0.1:38918)
0928 08:42:55.389226 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.389342 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 08:42:55.390614 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.390191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44819 (local address 127.0.0.1:38922)
0928 08:42:55.390343 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.390425 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 08:42:55.390810 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44819: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44819: connect: Connection refused (error 111)
I20260928 08:42:55.482100 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } attempt: 3
I20260928 08:42:55.483560 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.482956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44819 (local address 127.0.0.1:38932)
0928 08:42:55.483231 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.483378 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 08:42:55.484678 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.484214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44819 (local address 127.0.0.1:38942)
0928 08:42:55.484386 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.484488 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 08:42:55.484937 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44819: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44819: connect: Connection refused (error 111)
W20260928 08:42:55.615156 22859 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:42:55.615638 22859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:55.615778 22859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:55.615936 22859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:55.623152 22859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:55.623229 22859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:55.623304 22859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:55.623353 22859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:55.623406 22859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:55.628213 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } attempt: 4
I20260928 08:42:55.629469 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.628873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44819 (local address 127.0.0.1:38958)
0928 08:42:55.629185 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.629306 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260928 08:42:55.630574 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.630158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44819 (local address 127.0.0.1:38972)
0928 08:42:55.630311 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.630407 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 08:42:55.630753 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44819: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44819: connect: Connection refused (error 111)
I20260928 08:42:55.633612 22859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34695,127.0.97.61:44819,127.0.97.60:42125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:55.634850 22859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:55.636034 22859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:42:55.647425 22867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:55.647387 22864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:55.647691 22865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:55.648294 22859 server_base.cc:1034] running on GCE node
I20260928 08:42:55.648989 22859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:55.650238 22859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:55.651477 22859 hybrid_clock.cc:648] HybridClock initialized: now 1790584975651399 us; error 53 us; skew 500 ppm
I20260928 08:42:55.651795 22859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:55 dist-test-slave-w6c0 krb5kdc[22772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:55.654831 22859 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260928 08:42:55.656212 22859 webserver.cc:460] Webserver started at http://127.0.97.61:45925/ using document root <none> and password file <none>
I20260928 08:42:55.656885 22859 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:55.657016 22859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:55.657320 22859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:55.660197 22859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-1/data/instance:
uuid: "e3146d1e47ad4cfc84f718204cef63d9"
format_stamp: "Formatted at 2026-09-28 08:42:55 on dist-test-slave-w6c0"
server_key: "d35a84f8c4429e90090ca6806d9cb99a"
server_key_iv: "fff3cd71021c83f560638dd45e5d09ba"
server_key_version: "encryptionkey@0"
I20260928 08:42:55.660984 22859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-1/wal/instance:
uuid: "e3146d1e47ad4cfc84f718204cef63d9"
format_stamp: "Formatted at 2026-09-28 08:42:55 on dist-test-slave-w6c0"
server_key: "d35a84f8c4429e90090ca6806d9cb99a"
server_key_iv: "fff3cd71021c83f560638dd45e5d09ba"
server_key_version: "encryptionkey@0"
I20260928 08:42:55.665781 22859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 08:42:55.669477 22874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:55.670776 22859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:42:55.670955 22859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-1/wal
uuid: "e3146d1e47ad4cfc84f718204cef63d9"
format_stamp: "Formatted at 2026-09-28 08:42:55 on dist-test-slave-w6c0"
server_key: "d35a84f8c4429e90090ca6806d9cb99a"
server_key_iv: "fff3cd71021c83f560638dd45e5d09ba"
server_key_version: "encryptionkey@0"
I20260928 08:42:55.671170 22859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:55.685338 22859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:55.689072 22859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:55.689350 22859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:55.707445 22859 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44819
I20260928 08:42:55.707463 22925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44819 every 8 connection(s)
I20260928 08:42:55.708793 22859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-1/data/info.pb
I20260928 08:42:55.712347 22926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:55.713847 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 22859
I20260928 08:42:55.714023 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-1/wal/instance
I20260928 08:42:55.714725 388 external_mini_cluster.cc:1428] Setting key f970aed2ee68b4ba23268caa47b693b0
I20260928 08:42:55.721801 22926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:42:55 dist-test-slave-w6c0 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260928 08:42:55.742620 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.725170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.0.97.62:34695)
0928 08:42:55.725405 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:55.725410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:55.726529 (+ 1119us) server_negotiation.cc:407] Connection header received
0928 08:42:55.728767 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:55.728776 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:55.728878 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:55.728967 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:55.730623 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:55.731304 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:55.732751 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:55.733186 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:55.733342 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:55.736728 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:55.736736 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:55.736752 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:55.736793 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:55.739527 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:55.740223 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:55.740236 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:55.740240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:55.740367 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:55.740792 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:55.740799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:55.740802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:55.741076 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:55.741255 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:55.742004 (+ 749us) server_negotiation.cc:299] Negotiation successful
0928 08:42:55.742232 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260928 08:42:55.743656 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.725609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34695 (local address 127.0.0.1:48318)
0928 08:42:55.726356 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.726393 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:55.728376 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:55.729363 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:55.729370 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:55.729865 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:55.730399 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:55.730414 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:55.731461 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:55.731472 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:55.732607 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:55.732618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:55.733420 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:55.733427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:55.733648 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:55.734333 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:42:55.734356 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:55.736481 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:55.739671 (+ 3190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:55.739679 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:55.739694 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:55.740096 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:55.740480 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:55.740486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:55.740490 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:55.740673 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:55.741212 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:55.741221 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:55.741647 (+ 426us) client_negotiation.cc:769] Sending connection context
0928 08:42:55.741912 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 08:42:55.742474 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":318,"threads_started":1}
I20260928 08:42:55.745429 22926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:42:55 dist-test-slave-w6c0 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 08:42:55.760064 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.746382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38978 (local address 127.0.97.61:44819)
0928 08:42:55.746941 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:55.746947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:55.746969 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:42:55.747007 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:55.747012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:55.747168 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:55.747320 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:55.748004 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:55.749171 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:55.750252 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:55.750848 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:55.750980 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:55.754341 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:55.754351 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:55.754364 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:55.754391 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:55.757163 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:55.757965 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:55.757971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:55.757973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:55.758075 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:55.758550 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:55.758557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:55.758565 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:55.758867 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:55.759010 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:55.759449 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 08:42:55.759779 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":423,"thread_start_us":276,"threads_started":1}
I20260928 08:42:55.760139 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.746264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44819 (local address 127.0.0.1:38978)
0928 08:42:55.746449 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.746465 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:55.746601 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:55.747307 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:55.747311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:55.747719 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:55.747837 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:55.747844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:55.749319 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:55.749327 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:55.750124 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:55.750133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:55.750983 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:55.750988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:55.751174 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:55.751721 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:42:55.751737 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:55.754129 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:55.757293 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:55.757301 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:55.757308 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:55.757845 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:55.758197 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:55.758205 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:55.758211 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:55.758389 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:55.759000 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:55.759008 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:55.759176 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 08:42:55.759334 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 08:42:55.759663 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 08:42:55.762624 22926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:42:55.763748 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.763282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43020)
0928 08:42:55.763427 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.763596 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 08:42:55.764066 22876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
I20260928 08:42:55.765573 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.765278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43034)
0928 08:42:55.765404 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.765475 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 08:42:55.765879 22926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42125: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260928 08:42:55.773485 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:42125
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34695,127.0.97.61:44819,127.0.97.60:42125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:42:55.815182 22926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } attempt: 1
I20260928 08:42:55.816406 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.815971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43036)
0928 08:42:55.816165 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.816251 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 08:42:55.817452 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.817136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43046)
0928 08:42:55.817289 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.817356 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 08:42:55.817749 22926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42125: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
I20260928 08:42:55.885967 22926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } attempt: 2
I20260928 08:42:55.887167 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.886717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43052)
0928 08:42:55.886910 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.887009 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 08:42:55.888173 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.887868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43062)
0928 08:42:55.887994 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.888059 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 08:42:55.888422 22926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42125: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
I20260928 08:42:55.930058 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } attempt: 5
Sep 28 08:42:55 dist-test-slave-w6c0 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 08:42:55.944900 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.930987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44819 (local address 127.0.0.1:38990)
0928 08:42:55.931202 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.931229 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:55.931380 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:55.931763 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:55.931770 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:55.932231 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:55.932407 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:55.932417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:55.933933 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:55.933936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:55.934891 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:55.934903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:55.935895 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:55.935900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:55.936044 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:55.936715 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:42:55.936733 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:55.938980 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:55.942712 (+ 3732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:55.942716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:55.942719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:55.943072 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:55.943485 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:55.943488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:55.943490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:55.943612 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:55.944213 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:55.944216 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:55.944337 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 08:42:55.944441 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 08:42:55.944638 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 08:42:55.945328 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.931156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38990 (local address 127.0.97.61:44819)
0928 08:42:55.931380 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:55.931387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:55.931410 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:42:55.931539 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:55.931544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:55.931613 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:55.931724 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:55.932608 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:55.933773 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:55.935060 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:55.935721 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:55.935857 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:55.939182 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:55.939189 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:55.939193 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:55.939231 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:55.942555 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:55.943206 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:55.943213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:55.943219 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:55.943373 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:55.943713 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:55.943718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:55.943722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:55.944086 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:55.944278 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:55.944607 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 08:42:55.944881 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 08:42:55.946295 22852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:42:55.947293 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.946900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43078)
0928 08:42:55.947056 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.947155 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 08:42:55.948220 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.947902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43086)
0928 08:42:55.948045 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.948118 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 08:42:55.948421 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42125: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
I20260928 08:42:55.979638 22926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } attempt: 3
I20260928 08:42:55.981081 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.980448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43100)
0928 08:42:55.980694 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.980841 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 08:42:55.982391 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:55.981968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43104)
0928 08:42:55.982192 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:55.982290 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260928 08:42:55.982615 22926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42125: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
I20260928 08:42:55.999675 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } attempt: 1
I20260928 08:42:56.000846 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.000349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43120)
0928 08:42:56.000611 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:56.000696 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260928 08:42:56.001853 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.001517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43124)
0928 08:42:56.001667 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:56.001743 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 08:42:56.002060 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42125: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
I20260928 08:42:56.070346 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } attempt: 2
I20260928 08:42:56.071558 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.071088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43130)
0928 08:42:56.071295 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:56.071402 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 08:42:56.072566 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.072258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43140)
0928 08:42:56.072398 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:56.072464 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 08:42:56.072755 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42125: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
W20260928 08:42:56.101897 22933 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:42:56.102269 22933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:42:56.102419 22933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:42:56.102579 22933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:42:56.109666 22933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:42:56.109784 22933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:42:56.109892 22933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:42:56.109982 22933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:42:56.110070 22933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:42:56.119603 22933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34695,127.0.97.61:44819,127.0.97.60:42125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:42125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:42:56.120906 22933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:42:56.122164 22933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:42:56.125871 22926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } attempt: 4
I20260928 08:42:56.127323 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.126664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43154)
0928 08:42:56.126945 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:56.127082 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 08:42:56.128631 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.128180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43158)
0928 08:42:56.128416 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:56.128502 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260928 08:42:56.128862 22926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42125: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
W20260928 08:42:56.133839 22938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:56.133780 22939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:42:56.133894 22941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:42:56.134820 22933 server_base.cc:1034] running on GCE node
I20260928 08:42:56.135438 22933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:42:56.136626 22933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:42:56.137841 22933 hybrid_clock.cc:648] HybridClock initialized: now 1790584976137781 us; error 32 us; skew 500 ppm
I20260928 08:42:56.138157 22933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:42:56 dist-test-slave-w6c0 krb5kdc[22772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:42:56.141208 22933 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260928 08:42:56.142652 22933 webserver.cc:460] Webserver started at http://127.0.97.60:34591/ using document root <none> and password file <none>
I20260928 08:42:56.143306 22933 fs_manager.cc:359] Metadata directory not provided
I20260928 08:42:56.143412 22933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:42:56.143733 22933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:42:56.146600 22933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-2/data/instance:
uuid: "e8e7b69286294a808ef6315f0025d4b0"
format_stamp: "Formatted at 2026-09-28 08:42:56 on dist-test-slave-w6c0"
server_key: "3e5ed11bea0b8022af16bf19f59359b3"
server_key_iv: "23d36b03a7c1636a7b59a218a9e960b3"
server_key_version: "encryptionkey@0"
I20260928 08:42:56.147368 22933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-2/wal/instance:
uuid: "e8e7b69286294a808ef6315f0025d4b0"
format_stamp: "Formatted at 2026-09-28 08:42:56 on dist-test-slave-w6c0"
server_key: "3e5ed11bea0b8022af16bf19f59359b3"
server_key_iv: "23d36b03a7c1636a7b59a218a9e960b3"
server_key_version: "encryptionkey@0"
I20260928 08:42:56.152108 22933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 08:42:56.155267 22948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:42:56.156461 22933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260928 08:42:56.156633 22933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-2/wal
uuid: "e8e7b69286294a808ef6315f0025d4b0"
format_stamp: "Formatted at 2026-09-28 08:42:56 on dist-test-slave-w6c0"
server_key: "3e5ed11bea0b8022af16bf19f59359b3"
server_key_iv: "23d36b03a7c1636a7b59a218a9e960b3"
server_key_version: "encryptionkey@0"
I20260928 08:42:56.156850 22933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:42:56.178942 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } attempt: 3
I20260928 08:42:56.180022 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.179550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43164)
0928 08:42:56.179775 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:56.179857 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 08:42:56.181274 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.180697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43166)
0928 08:42:56.180941 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:56.181104 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260928 08:42:56.181550 22852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42125: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42125: connect: Connection refused (error 111)
I20260928 08:42:56.188978 22933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:42:56.192759 22933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:42:56.193006 22933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:42:56.211320 22933 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:42125
I20260928 08:42:56.211364 22999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:42125 every 8 connection(s)
I20260928 08:42:56.212620 22933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-2/data/info.pb
I20260928 08:42:56.216406 23000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:42:56.217639 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 22933
I20260928 08:42:56.217850 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584618551208-388-0/minicluster-data/master-2/wal/instance
I20260928 08:42:56.218406 388 external_mini_cluster.cc:1428] Setting key 1474fb31c021aa08853c9533dfb97399
I20260928 08:42:56.227380 23000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:42:56 dist-test-slave-w6c0 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 28 08:42:56 dist-test-slave-w6c0 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 08:42:56.248287 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.231479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48330 (local address 127.0.97.62:34695)
0928 08:42:56.231695 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.231701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.232643 (+ 942us) server_negotiation.cc:407] Connection header received
0928 08:42:56.234189 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.234194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.234279 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.234397 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.235649 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.236367 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.238073 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.238566 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.238702 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:56.242226 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:56.242233 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:56.242243 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:56.242285 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:56.245040 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.245874 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.245878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.245880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.246010 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.246487 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.246491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.246494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.246894 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:56.247059 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:56.247808 (+ 749us) server_negotiation.cc:299] Negotiation successful
0928 08:42:56.247978 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 08:42:56.249835 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.220330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.0.97.60:42125)
0928 08:42:56.231062 (+ 10732us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.231076 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.231719 (+ 643us) server_negotiation.cc:407] Connection header received
0928 08:42:56.233877 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.233899 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.234110 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.234322 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.235792 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.236917 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.238134 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.238717 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.238988 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:56.243187 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:56.243194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:56.243206 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:56.243234 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:56.246351 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.247165 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.247172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.247175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.247327 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.247676 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.247682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.247688 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.248095 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:56.248400 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:56.248738 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 08:42:56.248962 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":10512,"thread_start_us":10296,"threads_started":1}
I20260928 08:42:56.249873 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.231778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34695 (local address 127.0.0.1:48330)
0928 08:42:56.232490 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:56.232534 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:56.233911 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:56.234469 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:56.234475 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:56.235115 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:56.235476 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:56.235493 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.236524 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.236536 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:56.237908 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:56.237920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.238793 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.238803 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:56.239029 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:56.239684 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:42:56.239705 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:56.242018 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:56.245168 (+ 3150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:56.245179 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:56.245192 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:56.245755 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:56.246116 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:56.246121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:56.246125 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:56.246379 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:56.247027 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:56.247033 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:56.247441 (+ 408us) client_negotiation.cc:769] Sending connection context
0928 08:42:56.247725 (+ 284us) client_negotiation.cc:240] Negotiation successful
0928 08:42:56.248197 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":5,"spinlock_wait_cycles":12544,"thread_start_us":289,"threads_started":1}
I20260928 08:42:56.251681 23000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:42:56 dist-test-slave-w6c0 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 28 08:42:56 dist-test-slave-w6c0 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:42:56.265316 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.252557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44819 (local address 127.0.0.1:39000)
0928 08:42:56.252695 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:56.252723 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:56.252849 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:56.253343 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:56.253347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:56.253638 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:56.253762 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:56.253769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.254765 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.254772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:56.255635 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:56.255645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.256349 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.256355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:56.256527 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:56.257072 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:42:56.257088 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:56.259118 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:56.262918 (+ 3800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:56.262921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:56.262923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:56.263269 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:56.263678 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:56.263681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:56.263683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:56.263811 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:56.264480 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:56.264487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:56.264647 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 08:42:56.264811 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 08:42:56.265093 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 08:42:56.266227 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.252511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39000 (local address 127.0.97.61:44819)
0928 08:42:56.252697 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.252704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.252799 (+ 95us) server_negotiation.cc:407] Connection header received
0928 08:42:56.252971 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.252977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.253070 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.253164 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.253901 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.254625 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.255780 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.256208 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.256344 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:56.259299 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:56.259305 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:56.259309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:56.259349 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:56.262797 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.263391 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.263398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.263402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.263565 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.263919 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.263927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.263933 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.264266 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:56.264415 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:56.265633 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0928 08:42:56.265874 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:42:56.267186 23000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:42:56.267902 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.253146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48332 (local address 127.0.97.62:34695)
0928 08:42:56.253331 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.253336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.253353 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:42:56.253444 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.253449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.253515 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.253612 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.254727 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.255440 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.256630 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.257081 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.257213 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:56.261435 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:56.261441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:56.261446 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:56.261489 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:56.264895 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.265711 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.265716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.265719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.265861 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.266191 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.266199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.266203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.266592 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:56.266833 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:56.267365 (+ 532us) server_negotiation.cc:299] Negotiation successful
0928 08:42:56.267576 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 08:42:56 dist-test-slave-w6c0 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 28 08:42:56 dist-test-slave-w6c0 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 08:42:56.280941 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.268053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.0.97.60:42125)
0928 08:42:56.268192 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.268196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.268210 (+ 14us) server_negotiation.cc:407] Connection header received
0928 08:42:56.268247 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.268250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.268297 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.268421 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.269243 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.270265 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.271540 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.271933 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.272030 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:56.275766 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:56.275770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:56.275774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:56.275800 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:56.278220 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.279015 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.279023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.279027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.279154 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.279507 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.279512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.279515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.279790 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:56.279935 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:56.280270 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 08:42:56.280532 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":33}
I20260928 08:42:56.281150 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.267861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43186)
0928 08:42:56.268036 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:56.268054 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:56.268143 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:56.268579 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:56.268582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:56.268979 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:56.269106 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:56.269114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.270400 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.270404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:56.271404 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:56.271417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.272084 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.272089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:56.272311 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:56.272865 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:42:56.272882 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:56.275549 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:56.278346 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:56.278353 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:56.278357 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:56.278897 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:56.279260 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:56.279267 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:56.279273 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:56.279401 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:56.279899 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:56.279903 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:56.280054 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 08:42:56.280184 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 08:42:56.280673 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":170}
I20260928 08:42:56.283713 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.269513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39010 (local address 127.0.97.61:44819)
0928 08:42:56.269653 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.269656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.269671 (+ 15us) server_negotiation.cc:407] Connection header received
0928 08:42:56.269807 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.269811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.269860 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.269932 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.271242 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.272076 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.273191 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.273865 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.274011 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:56.278095 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:56.278101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:56.278105 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:56.278146 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:56.280770 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.281549 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.281552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.281555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.281738 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.282172 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.282178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.282182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.282551 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:56.282831 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:56.283135 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 08:42:56.283367 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 08:42:56.286759 23000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0: Bootstrap starting.
I20260928 08:42:56.288607 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:42:56.290242 23000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:56.291450 23000 log.cc:824] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:56.291333 22827 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "07bd497d0c7044888fecd45b6cb77296"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48332
I20260928 08:42:56.294193 23000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0: No bootstrap required, opened a new log
I20260928 08:42:56.301252 23000 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd497d0c7044888fecd45b6cb77296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } } peers { permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } } peers { permanent_uuid: "e8e7b69286294a808ef6315f0025d4b0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } }
I20260928 08:42:56.301595 23000 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e7b69286294a808ef6315f0025d4b0, State: Initialized, Role: FOLLOWER
I20260928 08:42:56.302026 23000 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd497d0c7044888fecd45b6cb77296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } } peers { permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } } peers { permanent_uuid: "e8e7b69286294a808ef6315f0025d4b0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } }
I20260928 08:42:56.302932 23006 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd497d0c7044888fecd45b6cb77296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } } peers { permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } } peers { permanent_uuid: "e8e7b69286294a808ef6315f0025d4b0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } } }
I20260928 08:42:56.303264 23006 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:42:56.304189 23000 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 08:42:56.313946 23017 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:42:56.314059 23017 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:42:56.311687 22933 master_runner.cc:186] Error getting master registration for 127.0.97.62:34695: OK, instance_id { permanent_uuid: "07bd497d0c7044888fecd45b6cb77296" instance_seqno: 1790584975213237 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:42:56.319779 22933 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 08:42:56.358793 22852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } attempt: 4
Sep 28 08:42:56 dist-test-slave-w6c0 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 08:42:56.380452 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.359598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43188)
0928 08:42:56.359810 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:56.359832 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:56.359976 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:56.360354 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:56.360359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:56.360871 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:56.361084 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:56.361093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.362237 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.362246 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:56.363313 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:56.363325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.364131 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.364137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:56.364347 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:56.370091 (+ 5744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:42:56.370126 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:56.373195 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:56.377587 (+ 4392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:56.377592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:56.377595 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:56.377966 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:56.378385 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:56.378396 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:56.378400 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:56.378639 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:56.379391 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:56.379395 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:56.379572 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 08:42:56.379708 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 08:42:56.380019 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 08:42:56.380476 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.359673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43188 (local address 127.0.97.60:42125)
0928 08:42:56.359872 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.359880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.359904 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:42:56.360127 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.360133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.360226 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.360324 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.361237 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.362072 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.363457 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.363970 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.364107 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:56.373535 (+ 9428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:56.373542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:56.373548 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:56.373599 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:56.377440 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.378087 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.378093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.378097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.378254 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.378755 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.378760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.378764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.379266 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:56.379425 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:56.379859 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 08:42:56.380113 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 08:42:56.386478 22852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296: Bootstrap starting.
I20260928 08:42:56.389770 22852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:56.390908 22852 log.cc:824] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:56.393740 22852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296: No bootstrap required, opened a new log
I20260928 08:42:56.401389 22852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd497d0c7044888fecd45b6cb77296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } } peers { permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } } peers { permanent_uuid: "e8e7b69286294a808ef6315f0025d4b0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } }
I20260928 08:42:56.401808 22852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bd497d0c7044888fecd45b6cb77296, State: Initialized, Role: FOLLOWER
I20260928 08:42:56.402307 22852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd497d0c7044888fecd45b6cb77296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } } peers { permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } } peers { permanent_uuid: "e8e7b69286294a808ef6315f0025d4b0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } }
I20260928 08:42:56.403373 23020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd497d0c7044888fecd45b6cb77296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } } peers { permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } } peers { permanent_uuid: "e8e7b69286294a808ef6315f0025d4b0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } } }
I20260928 08:42:56.403738 23020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:42:56.404610 22852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:56.404722 22827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 08:42:56.404939 22827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:56.408504 22827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd497d0c7044888fecd45b6cb77296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } } peers { permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } } peers { permanent_uuid: "e8e7b69286294a808ef6315f0025d4b0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } }
I20260928 08:42:56.411378 22975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07bd497d0c7044888fecd45b6cb77296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e8e7b69286294a808ef6315f0025d4b0"
I20260928 08:42:56.411867 22975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:56.414750 22827 leader_election.cc:288] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [CANDIDATE]: Term 1 election: Requested vote from peers e3146d1e47ad4cfc84f718204cef63d9 (127.0.97.61:44819), e8e7b69286294a808ef6315f0025d4b0 (127.0.97.60:42125)
I20260928 08:42:56.415447 22975 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07bd497d0c7044888fecd45b6cb77296 in term 1.
I20260928 08:42:56.418241 22802 leader_election.cc:302] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07bd497d0c7044888fecd45b6cb77296, e8e7b69286294a808ef6315f0025d4b0; no voters:
I20260928 08:42:56.418743 23020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:42:56.420936 23020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [term 1 LEADER]: Becoming Leader. State: Replica: 07bd497d0c7044888fecd45b6cb77296, State: Running, Role: LEADER
I20260928 08:42:56.421584 23020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd497d0c7044888fecd45b6cb77296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } } peers { permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } } peers { permanent_uuid: "e8e7b69286294a808ef6315f0025d4b0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } }
I20260928 08:42:56.428063 22926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } attempt: 5
Sep 28 08:42:56 dist-test-slave-w6c0 krb5kdc[22772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 08:42:56.440387 23027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07bd497d0c7044888fecd45b6cb77296. Latest consensus state: current_term: 1 leader_uuid: "07bd497d0c7044888fecd45b6cb77296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd497d0c7044888fecd45b6cb77296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } } peers { permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } } peers { permanent_uuid: "e8e7b69286294a808ef6315f0025d4b0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } } }
I20260928 08:42:56.440850 23027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [sys.catalog]: This master's current role is: LEADER
W20260928 08:42:56.445089 22954 tablet.cc:2369] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:42:56.451021 22806 tablet.cc:2369] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:42:56.451412 23032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:42:56.454178 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.428790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42125 (local address 127.0.0.1:43192)
0928 08:42:56.429078 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 08:42:56.429098 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:42:56.429278 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:42:56.429779 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:42:56.429787 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:42:56.430447 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:42:56.430770 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:56.430789 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.432914 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.432922 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:56.434830 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:42:56.434849 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.436400 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.436409 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:42:56.436735 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:56.437600 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:42:56.437632 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:42:56.441898 (+ 4266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:42:56.451159 (+ 9261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:56.451165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:56.451169 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:56.451769 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:56.452264 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:42:56.452270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:42:56.452273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:42:56.452450 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:42:56.453146 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:42:56.453152 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:42:56.453322 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 08:42:56.453466 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 08:42:56.453795 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260928 08:42:56.455410 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.428977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.0.97.60:42125)
0928 08:42:56.429151 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.429157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.429176 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:42:56.429427 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.429438 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.429563 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.430961 (+ 1398us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.431153 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.432548 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.435218 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.436188 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.442081 (+ 5893us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:56.442301 (+ 220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:56.442308 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:56.442313 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:56.442341 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:56.448767 (+ 6426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.451938 (+ 3171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.451945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.451949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.452122 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.452603 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.452609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.452614 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.452971 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:56.454579 (+ 1608us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:56.454715 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 08:42:56.454989 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:42:56.461927 22926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9: Bootstrap starting.
I20260928 08:42:56.466301 22926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9: Neither blocks nor log segments found. Creating new log.
I20260928 08:42:56.467710 22926 log.cc:824] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9: Log is configured to *not* fsync() on all Append() calls
I20260928 08:42:56.469619 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.442670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48342 (local address 127.0.97.62:34695)
0928 08:42:56.442871 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.442877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.443227 (+ 350us) server_negotiation.cc:407] Connection header received
0928 08:42:56.443679 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.443685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.443763 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.443836 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.447840 (+ 4004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.448970 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.453785 (+ 4815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.454480 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.454601 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:56.460580 (+ 5979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:56.460586 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:56.460590 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:56.460634 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:56.464662 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.465996 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.466002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.466005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.466190 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.467071 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.467076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.467079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.467491 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:56.467745 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:56.468792 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0928 08:42:56.469097 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:42:56.475478 22926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9: No bootstrap required, opened a new log
I20260928 08:42:56.491634 23032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:42:56.500747 22926 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd497d0c7044888fecd45b6cb77296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } } peers { permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } } peers { permanent_uuid: "e8e7b69286294a808ef6315f0025d4b0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } }
I20260928 08:42:56.501813 22926 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3146d1e47ad4cfc84f718204cef63d9, State: Initialized, Role: FOLLOWER
I20260928 08:42:56.502995 22926 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd497d0c7044888fecd45b6cb77296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } } peers { permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } } peers { permanent_uuid: "e8e7b69286294a808ef6315f0025d4b0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } }
I20260928 08:42:56.509370 23042 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd497d0c7044888fecd45b6cb77296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } } peers { permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } } peers { permanent_uuid: "e8e7b69286294a808ef6315f0025d4b0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } } }
I20260928 08:42:56.510727 23042 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:42:56.512565 22901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07bd497d0c7044888fecd45b6cb77296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e3146d1e47ad4cfc84f718204cef63d9"
I20260928 08:42:56.513116 22901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:42:56.517252 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.443920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39014 (local address 127.0.97.61:44819)
0928 08:42:56.444083 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.444088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.468640 (+ 24552us) server_negotiation.cc:407] Connection header received
0928 08:42:56.469759 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.469776 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.469857 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.469927 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.483108 (+ 13181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.484312 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.495431 (+ 11119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.496189 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.496292 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:56.504520 (+ 8228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:56.504527 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:56.504532 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:56.504578 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:56.508570 (+ 3992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.509653 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.509659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.509663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.509852 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.510974 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.510979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.510983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.511383 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:56.511647 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:56.516529 (+ 4882us) server_negotiation.cc:299] Negotiation successful
0928 08:42:56.516806 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:42:56.518055 22901 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07bd497d0c7044888fecd45b6cb77296 in term 1.
I20260928 08:42:56.518585 22926 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:42:56.523592 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:56.455050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.0.97.60:42125)
0928 08:42:56.467964 (+ 12914us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.467974 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.468000 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:42:56.468055 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.468061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.468135 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.468244 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.473104 (+ 4860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.484772 (+ 11668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.488431 (+ 3659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:56.489143 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:56.489284 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:56.501483 (+ 12199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:42:56.501490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:42:56.501495 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:42:56.501540 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:42:56.508093 (+ 6553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.510296 (+ 2203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.510302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.510305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.510521 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:42:56.515864 (+ 5343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:42:56.515870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:42:56.515874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:42:56.520114 (+ 4240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:42:56.520362 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:56.522956 (+ 2594us) server_negotiation.cc:299] Negotiation successful
0928 08:42:56.523206 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12753,"thread_start_us":13391,"threads_started":1}
I20260928 08:42:56.534792 22975 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0 [term 1 FOLLOWER]: Refusing update from remote peer 07bd497d0c7044888fecd45b6cb77296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:42:56.536131 23027 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8e7b69286294a808ef6315f0025d4b0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:42:56.548112 22901 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9 [term 1 FOLLOWER]: Refusing update from remote peer 07bd497d0c7044888fecd45b6cb77296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:42:56.549311 23027 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:42:56.559437 23042 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07bd497d0c7044888fecd45b6cb77296. Latest consensus state: current_term: 1 leader_uuid: "07bd497d0c7044888fecd45b6cb77296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd497d0c7044888fecd45b6cb77296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } } peers { permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } } peers { permanent_uuid: "e8e7b69286294a808ef6315f0025d4b0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } } }
I20260928 08:42:56.559825 23042 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:42:56.565285 23044 mvcc.cc:205] Tried to move back new op lower bound from 7334236063849385984 to 7334236063430037504. Current Snapshot: MvccSnapshot[applied={T|T < 7334236063849385984}]
I20260928 08:42:56.571563 23042 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07bd497d0c7044888fecd45b6cb77296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd497d0c7044888fecd45b6cb77296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } } peers { permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } } peers { permanent_uuid: "e8e7b69286294a808ef6315f0025d4b0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } } }
I20260928 08:42:56.571904 23042 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:42:56.573822 23027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07bd497d0c7044888fecd45b6cb77296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd497d0c7044888fecd45b6cb77296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } } peers { permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } } peers { permanent_uuid: "e8e7b69286294a808ef6315f0025d4b0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } } }
I20260928 08:42:56.574077 23027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:56.574766 23051 mvcc.cc:205] Tried to move back new op lower bound from 7334236063849385984 to 7334236063430037504. Current Snapshot: MvccSnapshot[applied={T|T < 7334236063849385984 or (T in {7334236063849385984})}]
I20260928 08:42:56.575388 23006 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07bd497d0c7044888fecd45b6cb77296. Latest consensus state: current_term: 1 leader_uuid: "07bd497d0c7044888fecd45b6cb77296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd497d0c7044888fecd45b6cb77296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } } peers { permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } } peers { permanent_uuid: "e8e7b69286294a808ef6315f0025d4b0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } } }
I20260928 08:42:56.575685 23006 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:42:56.580390 23049 mvcc.cc:205] Tried to move back new op lower bound from 7334236063849385984 to 7334236063430037504. Current Snapshot: MvccSnapshot[applied={T|T < 7334236063849385984}]
I20260928 08:42:56.584524 23020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07bd497d0c7044888fecd45b6cb77296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd497d0c7044888fecd45b6cb77296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } } peers { permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } } peers { permanent_uuid: "e8e7b69286294a808ef6315f0025d4b0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } } }
I20260928 08:42:56.585851 23020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296 [sys.catalog]: This master's current role is: LEADER
I20260928 08:42:56.585879 23032 catalog_manager.cc:1347] Generated new cluster ID: c4227a71693149c9a3048bf5ebd40d1c
I20260928 08:42:56.586335 23032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:42:56.579891 22785 master_runner.cc:186] Error getting master registration for 127.0.97.61:44819: OK, instance_id { permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" instance_seqno: 1790584975702247 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:42:56.594713 23006 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07bd497d0c7044888fecd45b6cb77296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd497d0c7044888fecd45b6cb77296" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34695 } } peers { permanent_uuid: "e3146d1e47ad4cfc84f718204cef63d9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44819 } } peers { permanent_uuid: "e8e7b69286294a808ef6315f0025d4b0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42125 } } }
I20260928 08:42:56.594982 23006 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:42:56.597862 22785 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 08:42:56.625855 23065 catalog_manager.cc:1259] Loaded cluster ID: c4227a71693149c9a3048bf5ebd40d1c
I20260928 08:42:56.626045 23065 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9: loading cluster ID for follower catalog manager: success
I20260928 08:42:56.626637 23032 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:42:56.628364 23032 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:42:56.629742 23065 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9: acquiring CA information for follower catalog manager: success
W20260928 08:42:56.630928 23065 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260928 08:42:56.649840 23032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07bd497d0c7044888fecd45b6cb77296: Generated new TSK 0
I20260928 08:42:56.651036 23032 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 08:42:56.795866 22856 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:42:56.768938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.0.97.62:34695)
0928 08:42:56.769154 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.769160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.791437 (+ 22277us) server_negotiation.cc:407] Connection header received
0928 08:42:56.794225 (+ 2788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.794233 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.794327 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.794407 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.795601 (+ 1194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48348: BlockingRecv error: recv got EOF from 127.0.0.1:48348 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 08:42:56.800830 22930 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:42:56.775963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39022 (local address 127.0.97.61:44819)
0928 08:42:56.776161 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.776168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.797850 (+ 21682us) server_negotiation.cc:407] Connection header received
0928 08:42:56.799263 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.799270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.799372 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.799440 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.800557 (+ 1117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39022: BlockingRecv error: recv got EOF from 127.0.0.1:39022 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 08:42:56.802789 23037 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:42:56.792068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43208 (local address 127.0.97.60:42125)
0928 08:42:56.792267 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.792274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.798999 (+ 6725us) server_negotiation.cc:407] Connection header received
0928 08:42:56.801395 (+ 2396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.801402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.801495 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.801574 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.802541 (+ 967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43208: BlockingRecv error: recv got EOF from 127.0.0.1:43208 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 08:42:56.847919 22930 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:42:56.834440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.0.97.61:44819)
0928 08:42:56.834685 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.834691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.845736 (+ 11045us) server_negotiation.cc:407] Connection header received
0928 08:42:56.846290 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.846297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.846449 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.846551 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.847637 (+ 1086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39028: BlockingRecv error: recv got EOF from 127.0.0.1:39028 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260928 08:42:56.888650 22930 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:42:56.883674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.0.97.61:44819)
0928 08:42:56.883863 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.883869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.887096 (+ 3227us) server_negotiation.cc:407] Connection header received
0928 08:42:56.887322 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.887328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.887435 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.887553 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.888408 (+ 855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39042: BlockingRecv error: recv got EOF from 127.0.0.1:39042 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 08:42:56.902395 22930 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:42:56.900760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.0.97.61:44819)
0928 08:42:56.900958 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.900965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.900992 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:42:56.901068 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.901074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.901177 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.901299 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.902110 (+ 811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39058: BlockingRecv error: recv got EOF from 127.0.0.1:39058 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 08:42:56.912348 22930 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:42:56.910210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39068 (local address 127.0.97.61:44819)
0928 08:42:56.910424 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.910430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.910929 (+ 499us) server_negotiation.cc:407] Connection header received
0928 08:42:56.911103 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.911109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.911200 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.911321 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.912079 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39068: BlockingRecv error: recv got EOF from 127.0.0.1:39068 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260928 08:42:56.925622 22930 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:42:56.923500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.0.97.61:44819)
0928 08:42:56.923688 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.923694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.924168 (+ 474us) server_negotiation.cc:407] Connection header received
0928 08:42:56.924365 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.924372 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.924474 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.924589 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.925371 (+ 782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39080: BlockingRecv error: recv got EOF from 127.0.0.1:39080 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 08:42:56.950773 22930 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:42:56.948405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.0.97.61:44819)
0928 08:42:56.948591 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.948597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.949136 (+ 539us) server_negotiation.cc:407] Connection header received
0928 08:42:56.949343 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.949349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.949472 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.949584 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.950481 (+ 897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39094: BlockingRecv error: recv got EOF from 127.0.0.1:39094 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 08:42:56.990859 22930 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:42:56.988405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39104 (local address 127.0.97.61:44819)
0928 08:42:56.988627 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:56.988633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:56.989212 (+ 579us) server_negotiation.cc:407] Connection header received
0928 08:42:56.989429 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:56.989435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:56.989546 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:56.989666 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:56.990558 (+ 892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39104: BlockingRecv error: recv got EOF from 127.0.0.1:39104 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 08:42:57.063769 22930 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:42:57.061262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.0.97.61:44819)
0928 08:42:57.061480 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:57.061486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:57.062162 (+ 676us) server_negotiation.cc:407] Connection header received
0928 08:42:57.062366 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:57.062373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:57.062483 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:57.062628 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:57.063517 (+ 889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39114: BlockingRecv error: recv got EOF from 127.0.0.1:39114 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 08:42:57.202996 22930 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:42:57.200522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39126 (local address 127.0.97.61:44819)
0928 08:42:57.200749 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:57.200758 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:57.201315 (+ 557us) server_negotiation.cc:407] Connection header received
0928 08:42:57.201535 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:57.201541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:57.201638 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:57.201741 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:57.202786 (+ 1045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39126: BlockingRecv error: recv got EOF from 127.0.0.1:39126 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 08:42:57.316033 23017 catalog_manager.cc:1259] Loaded cluster ID: c4227a71693149c9a3048bf5ebd40d1c
I20260928 08:42:57.316140 23017 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0: loading cluster ID for follower catalog manager: success
I20260928 08:42:57.318790 23017 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0: acquiring CA information for follower catalog manager: success
I20260928 08:42:57.320674 23017 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e8e7b69286294a808ef6315f0025d4b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 08:42:57.466212 22930 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:42:57.464301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39138 (local address 127.0.97.61:44819)
0928 08:42:57.464587 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:57.464593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:57.464727 (+ 134us) server_negotiation.cc:407] Connection header received
0928 08:42:57.464972 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:57.464977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:57.465081 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:57.465195 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:42:57.466031 (+ 836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39138: BlockingRecv error: recv got EOF from 127.0.0.1:39138 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
I20260928 08:42:57.633668 23065 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e3146d1e47ad4cfc84f718204cef63d9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:42:57.988451 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:57.982565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39140 (local address 127.0.97.61:44819)
0928 08:42:57.983407 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:57.983412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:57.983429 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:42:57.983494 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:57.983497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:57.983706 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:57.983843 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:42:57.984292 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:57.985047 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:57.986321 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:57.986785 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:57.986878 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:57.987329 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:42:57.987832 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:42:57.987939 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:57.988054 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 08:42:57.988120 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":594,"threads_started":1}
I20260928 08:42:57.999632 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:57.993998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43224 (local address 127.0.97.60:42125)
0928 08:42:57.994455 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:57.994459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:57.994475 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:42:57.994720 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:57.994724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:57.994865 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:57.995002 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:42:57.995377 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:57.996063 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:57.997342 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:57.997758 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:57.997856 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:57.998355 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:42:57.998959 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:42:57.999133 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:57.999239 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 08:42:57.999307 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":244,"threads_started":1}
I20260928 08:42:58.012228 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:58.005921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39154 (local address 127.0.97.61:44819)
0928 08:42:58.006110 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:58.006115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:58.006445 (+ 330us) server_negotiation.cc:407] Connection header received
0928 08:42:58.006693 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:58.006699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:58.006802 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:58.006899 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:42:58.007569 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:58.008547 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:58.010405 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:58.010912 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:58.011052 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:58.011472 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:42:58.011568 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:42:58.011667 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:58.011826 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 08:42:58.011914 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 08:42:58.013677 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:58.006595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43226 (local address 127.0.97.60:42125)
0928 08:42:58.006738 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:58.006743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:58.007844 (+ 1101us) server_negotiation.cc:407] Connection header received
0928 08:42:58.007934 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:58.007938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:58.008015 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:58.008111 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:42:58.008461 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:58.009483 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:58.011566 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:58.012205 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:58.012310 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:58.012877 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:42:58.013029 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:42:58.013091 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:58.013323 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 08:42:58.013400 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 08:42:58.015646 23143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:42:58.005897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.0.97.62:34695)
0928 08:42:58.008868 (+ 2971us) server_negotiation.cc:206] Beginning negotiation
0928 08:42:58.008874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:42:58.008891 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:42:58.009001 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:42:58.009007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:42:58.009197 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:42:58.009376 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:42:58.010213 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:58.011318 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:58.013352 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:42:58.013973 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:42:58.014120 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:42:58.014665 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:42:58.015064 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:42:58.015157 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 08:42:58.015287 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 08:42:58.015350 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2848,"thread_start_us":1486,"threads_started":1}
I20260928 08:42:58.020068 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 22785
I20260928 08:42:58.021173 22847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:58.133877 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 22785
W20260928 08:42:58.154724 22875 connection.cc:535] server connection from 127.0.0.1:38990 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 08:42:58.155109 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 22859
I20260928 08:42:58.156112 22921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:58.264546 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 22859
I20260928 08:42:58.285343 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 22933
I20260928 08:42:58.286420 22995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:42:58.394917 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 22933
2026-09-28T08:42:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4979 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:42:58 dist-test-slave-w6c0 krb5kdc[23151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:42:58 dist-test-slave-w6c0 krb5kdc[23151](info): set up 2 sockets
Sep 28 08:42:58 dist-test-slave-w6c0 krb5kdc[23151](info): commencing operation
krb5kdc: starting...
W20260928 08:42:59.495106 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/test-user.keytab.
Sep 28 08:42:59 dist-test-slave-w6c0 krb5kdc[23151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:42:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:42:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:42:59.755606 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46241
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46241,127.0.97.61:42017,127.0.97.60:46129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:43:00.083029 23164 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:43:00.083329 23164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:43:00.083398 23164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:43:00.083479 23164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:43:00.090526 23164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:43:00.090616 23164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:43:00.090674 23164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:43:00.090724 23164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:43:00.090780 23164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:43:00.100426 23164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46241,127.0.97.61:42017,127.0.97.60:46129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:43:00.101712 23164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:43:00.103019 23164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:43:00.115017 23169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:00.115012 23172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:00.114955 23170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:43:00.115832 23164 server_base.cc:1034] running on GCE node
I20260928 08:43:00.116627 23164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:43:00.117957 23164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:43:00.119202 23164 hybrid_clock.cc:648] HybridClock initialized: now 1790584980119148 us; error 51 us; skew 500 ppm
I20260928 08:43:00.119532 23164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:43:00 dist-test-slave-w6c0 krb5kdc[23151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:43:00.122593 23164 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:43:00.124145 23164 webserver.cc:460] Webserver started at http://127.0.97.62:36183/ using document root <none> and password file <none>
I20260928 08:43:00.124825 23164 fs_manager.cc:359] Metadata directory not provided
I20260928 08:43:00.124948 23164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:43:00.125272 23164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:43:00.128203 23164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-0/data/instance:
uuid: "cf2ebd8da3484365bfb291aaf6dd701e"
format_stamp: "Formatted at 2026-09-28 08:43:00 on dist-test-slave-w6c0"
server_key: "dfcd244a5c42d0fdf8bbc67a2acad3c1"
server_key_iv: "36438e9ba488504b23298e0244afc84a"
server_key_version: "encryptionkey@0"
I20260928 08:43:00.129079 23164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-0/wal/instance:
uuid: "cf2ebd8da3484365bfb291aaf6dd701e"
format_stamp: "Formatted at 2026-09-28 08:43:00 on dist-test-slave-w6c0"
server_key: "dfcd244a5c42d0fdf8bbc67a2acad3c1"
server_key_iv: "36438e9ba488504b23298e0244afc84a"
server_key_version: "encryptionkey@0"
I20260928 08:43:00.134053 23164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:43:00.137378 23179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:00.138998 23164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:43:00.139165 23164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-0/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-0/wal
uuid: "cf2ebd8da3484365bfb291aaf6dd701e"
format_stamp: "Formatted at 2026-09-28 08:43:00 on dist-test-slave-w6c0"
server_key: "dfcd244a5c42d0fdf8bbc67a2acad3c1"
server_key_iv: "36438e9ba488504b23298e0244afc84a"
server_key_version: "encryptionkey@0"
I20260928 08:43:00.139343 23164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:43:00.164052 23164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:43:00.167893 23164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:43:00.168109 23164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:43:00.186976 23164 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46241
I20260928 08:43:00.187000 23230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46241 every 8 connection(s)
I20260928 08:43:00.188333 23164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:43:00.190167 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 23164
I20260928 08:43:00.190361 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-0/wal/instance
I20260928 08:43:00.190963 388 external_mini_cluster.cc:1428] Setting key f5e70e607668fad7d291ec5000e0f9eb
I20260928 08:43:00.192793 23231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:43:00.202720 23231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:43:00 dist-test-slave-w6c0 krb5kdc[23151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260928 08:43:00.227317 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.205977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33476 (local address 127.0.97.62:46241)
0928 08:43:00.207725 (+ 1748us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:00.207735 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:00.207788 (+ 53us) server_negotiation.cc:407] Connection header received
0928 08:43:00.209660 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:00.209671 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:00.209967 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:00.210182 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:00.211536 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:00.213480 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:00.214982 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:00.215980 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:00.216185 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:00.219615 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:00.219626 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:00.219641 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:00.219682 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:00.222981 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:00.223878 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:00.223884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:00.223887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:00.224097 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:00.224574 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:00.224577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:00.224579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:00.225064 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:00.225223 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:00.225905 (+ 682us) server_negotiation.cc:299] Negotiation successful
0928 08:43:00.226183 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1515,"spinlock_wait_cycles":61824,"thread_start_us":1311,"threads_started":1}
I20260928 08:43:00.227360 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.205978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46241 (local address 127.0.0.1:33476)
0928 08:43:00.207028 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.207095 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:00.209278 (+ 2183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:00.210202 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:00.210206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:00.210972 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:00.211306 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:00.211322 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:00.213648 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:00.213652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:00.214832 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:00.214844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:00.216217 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:00.216222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:00.216350 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:00.217119 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:00.217147 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:00.219347 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:00.223109 (+ 3762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:00.223124 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:00.223139 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:00.223737 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:00.224232 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:00.224240 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:00.224247 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:00.224457 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:00.225214 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:00.225225 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:00.225517 (+ 292us) client_negotiation.cc:769] Sending connection context
0928 08:43:00.225778 (+ 261us) client_negotiation.cc:240] Negotiation successful
0928 08:43:00.226346 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":75,"thread_start_us":481,"threads_started":1}
I20260928 08:43:00.230247 23231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:43:00.231544 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.231046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42017 (local address 127.0.0.1:50278)
0928 08:43:00.231247 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.231389 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 08:43:00.231837 23182 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42017: connect: Connection refused (error 111)
I20260928 08:43:00.233479 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.233037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42017 (local address 127.0.0.1:50292)
0928 08:43:00.233219 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.233321 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 08:43:00.233765 23231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42017: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42017: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260928 08:43:00.255391 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42017
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46241,127.0.97.61:42017,127.0.97.60:46129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:43:00.283113 23231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } attempt: 1
I20260928 08:43:00.284536 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.283874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42017 (local address 127.0.0.1:50294)
0928 08:43:00.284214 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.284363 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 08:43:00.285574 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.285247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42017 (local address 127.0.0.1:50304)
0928 08:43:00.285374 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.285443 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 08:43:00.285838 23231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42017: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42017: connect: Connection refused (error 111)
I20260928 08:43:00.354166 23231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } attempt: 2
I20260928 08:43:00.355531 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.355046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42017 (local address 127.0.0.1:50312)
0928 08:43:00.355291 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.355378 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 08:43:00.356516 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.356189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42017 (local address 127.0.0.1:50322)
0928 08:43:00.356325 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.356389 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 08:43:00.356736 23231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42017: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42017: connect: Connection refused (error 111)
I20260928 08:43:00.447984 23231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } attempt: 3
I20260928 08:43:00.449110 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.448677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42017 (local address 127.0.0.1:50326)
0928 08:43:00.448870 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.448971 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 08:43:00.450089 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.449740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42017 (local address 127.0.0.1:50338)
0928 08:43:00.449904 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.449973 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 08:43:00.450341 23231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42017: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42017: connect: Connection refused (error 111)
W20260928 08:43:00.586736 23238 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:43:00.587129 23238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:43:00.587253 23238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:43:00.587392 23238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 08:43:00.593601 23231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } attempt: 4
I20260928 08:43:00.594977 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.594333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42017 (local address 127.0.0.1:50342)
0928 08:43:00.594762 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.594849 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265}
W20260928 08:43:00.595264 23238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:43:00.595394 23238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:43:00.595515 23238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:43:00.595609 23238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:43:00.595695 23238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:43:00.596041 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.595675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42017 (local address 127.0.0.1:50356)
0928 08:43:00.595850 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.595926 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260928 08:43:00.596256 23231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42017: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42017: connect: Connection refused (error 111)
I20260928 08:43:00.605860 23238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46241,127.0.97.61:42017,127.0.97.60:46129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:43:00.607522 23238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:43:00.608979 23238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:43:00.620755 23243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:00.620795 23246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:00.620831 23244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:43:00.621397 23238 server_base.cc:1034] running on GCE node
I20260928 08:43:00.622268 23238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:43:00.623694 23238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:43:00.624915 23238 hybrid_clock.cc:648] HybridClock initialized: now 1790584980624836 us; error 49 us; skew 500 ppm
I20260928 08:43:00.625290 23238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:43:00 dist-test-slave-w6c0 krb5kdc[23151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:43:00.628517 23238 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260928 08:43:00.629976 23238 webserver.cc:460] Webserver started at http://127.0.97.61:42187/ using document root <none> and password file <none>
I20260928 08:43:00.630715 23238 fs_manager.cc:359] Metadata directory not provided
I20260928 08:43:00.630805 23238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:43:00.631076 23238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:43:00.634020 23238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-1/data/instance:
uuid: "189780c28f364bf3b705ad1e229e72c9"
format_stamp: "Formatted at 2026-09-28 08:43:00 on dist-test-slave-w6c0"
server_key: "f8130b3bc6c15d816dca2369ad6a9410"
server_key_iv: "b74c09fa2738ab9f3dd7b723d8590250"
server_key_version: "encryptionkey@0"
I20260928 08:43:00.634853 23238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-1/wal/instance:
uuid: "189780c28f364bf3b705ad1e229e72c9"
format_stamp: "Formatted at 2026-09-28 08:43:00 on dist-test-slave-w6c0"
server_key: "f8130b3bc6c15d816dca2369ad6a9410"
server_key_iv: "b74c09fa2738ab9f3dd7b723d8590250"
server_key_version: "encryptionkey@0"
I20260928 08:43:00.640014 23238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:43:00.643332 23253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:00.644649 23238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:43:00.644830 23238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-1/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-1/wal
uuid: "189780c28f364bf3b705ad1e229e72c9"
format_stamp: "Formatted at 2026-09-28 08:43:00 on dist-test-slave-w6c0"
server_key: "f8130b3bc6c15d816dca2369ad6a9410"
server_key_iv: "b74c09fa2738ab9f3dd7b723d8590250"
server_key_version: "encryptionkey@0"
I20260928 08:43:00.645059 23238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:43:00.688637 23238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:43:00.692808 23238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:43:00.693106 23238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:43:00.711824 23238 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42017
I20260928 08:43:00.711848 23304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42017 every 8 connection(s)
I20260928 08:43:00.713193 23238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-1/data/info.pb
I20260928 08:43:00.716862 23305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:43:00.720947 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 23238
I20260928 08:43:00.721143 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-1/wal/instance
I20260928 08:43:00.721712 388 external_mini_cluster.cc:1428] Setting key d2392111eceb77ab47e009438740be3a
I20260928 08:43:00.724942 23305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:43:00 dist-test-slave-w6c0 krb5kdc[23151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:43:00.747020 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.728208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33478 (local address 127.0.97.62:46241)
0928 08:43:00.728705 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:00.728714 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:00.729818 (+ 1104us) server_negotiation.cc:407] Connection header received
0928 08:43:00.731807 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:00.731816 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:00.731915 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:00.732049 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:00.733544 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:00.734260 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:00.735845 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:00.736263 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:00.736414 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:00.740312 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:00.740318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:00.740323 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:00.740371 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:00.743248 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:00.744335 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:00.744344 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:00.744350 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:00.744524 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:00.744952 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:00.744959 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:00.744965 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:00.745382 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:00.745572 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:00.746289 (+ 717us) server_negotiation.cc:299] Negotiation successful
0928 08:43:00.746571 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":222,"threads_started":1}
I20260928 08:43:00.747746 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.728697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46241 (local address 127.0.0.1:33478)
0928 08:43:00.729621 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.729676 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:00.731530 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:00.732235 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:00.732242 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:00.732776 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:00.733306 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:00.733334 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:00.734437 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:00.734441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:00.735676 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:00.735688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:00.736504 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:00.736511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:00.736737 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:00.737453 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:00.737479 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:00.740063 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:00.743397 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:00.743408 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:00.743429 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:00.744170 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:00.744645 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:00.744648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:00.744651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:00.744819 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:00.745516 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:00.745522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:00.745883 (+ 361us) client_negotiation.cc:769] Sending connection context
0928 08:43:00.746140 (+ 257us) client_negotiation.cc:240] Negotiation successful
0928 08:43:00.746613 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":358,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260928 08:43:00.749843 23305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 28 08:43:00 dist-test-slave-w6c0 krb5kdc[23151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 08:43:00.765559 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.750923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50364 (local address 127.0.97.61:42017)
0928 08:43:00.751647 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:00.751658 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:00.751690 (+ 32us) server_negotiation.cc:407] Connection header received
0928 08:43:00.751749 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:00.751761 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:00.752007 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:00.752161 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:00.752844 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:00.754570 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:00.755713 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:00.756631 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:00.756768 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:00.759810 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:00.759820 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:00.759833 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:00.759859 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:00.762648 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:00.763431 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:00.763437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:00.763439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:00.763547 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:00.763988 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:00.763991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:00.763993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:00.764326 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:00.764489 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:00.764940 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 08:43:00.765263 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":540,"thread_start_us":375,"threads_started":1}
I20260928 08:43:00.765647 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.750717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42017 (local address 127.0.0.1:50364)
0928 08:43:00.750964 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.750987 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:00.751096 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:00.752165 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:00.752169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:00.752527 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:00.752662 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:00.752670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:00.754725 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:00.754728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:00.755573 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:00.755584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:00.756767 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:00.756771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:00.756943 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:00.757556 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:00.757573 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:00.759601 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:00.762769 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:00.762778 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:00.762784 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:00.763300 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:00.763673 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:00.763681 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:00.763687 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:00.763881 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:00.764462 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:00.764469 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:00.764650 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 08:43:00.764808 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 08:43:00.765145 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":35}
I20260928 08:43:00.768203 23305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:43:00.769353 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.768788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:35862)
0928 08:43:00.769009 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.769155 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 08:43:00.769613 23256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
I20260928 08:43:00.771255 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.770830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:35878)
0928 08:43:00.771007 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.771112 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 08:43:00.771486 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46129: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260928 08:43:00.785390 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task38vn8d/build/asan/bin/kudu
/tmp/dist-test-task38vn8d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46129
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46241,127.0.97.61:42017,127.0.97.60:46129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:43:00.820845 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } attempt: 1
I20260928 08:43:00.822142 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.821584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:35886)
0928 08:43:00.821809 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.821951 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 08:43:00.823189 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.822793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:35888)
0928 08:43:00.822967 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.823056 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 08:43:00.823407 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46129: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
I20260928 08:43:00.891659 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } attempt: 2
I20260928 08:43:00.893131 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.892441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:35892)
0928 08:43:00.892712 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.892895 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 08:43:00.894228 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.893815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:35902)
0928 08:43:00.893976 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.894057 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 08:43:00.894484 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46129: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
I20260928 08:43:00.895462 23231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } attempt: 5
Sep 28 08:43:00 dist-test-slave-w6c0 krb5kdc[23151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 08:43:00.909432 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.896083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42017 (local address 127.0.0.1:50372)
0928 08:43:00.896285 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.896304 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:00.896434 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:00.896778 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:00.896781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:00.897139 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:00.897311 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:00.897320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:00.898710 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:00.898714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:00.899565 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:00.899576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:00.900565 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:00.900568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:00.900730 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:00.901352 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:00.901369 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:00.903547 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:00.907266 (+ 3719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:00.907270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:00.907273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:00.907633 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:00.908020 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:00.908023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:00.908025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:00.908143 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:00.908719 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:00.908721 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:00.908840 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 08:43:00.908956 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 08:43:00.909181 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 08:43:00.909623 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.896270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50372 (local address 127.0.97.61:42017)
0928 08:43:00.896436 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:00.896443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:00.896466 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:43:00.896561 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:00.896566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:00.896643 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:00.896742 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:00.897459 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:00.898570 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:00.899713 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:00.900419 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:00.900585 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:00.903748 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:00.903754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:00.903759 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:00.903798 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:00.907120 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:00.907752 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:00.907757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:00.907761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:00.907913 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:00.908243 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:00.908248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:00.908251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:00.908610 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:00.908759 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:00.909094 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 08:43:00.909317 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:43:00.910691 23231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:43:00.911619 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.911257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:35912)
0928 08:43:00.911413 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.911492 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 08:43:00.912467 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.912169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:35914)
0928 08:43:00.912297 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.912381 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 08:43:00.912676 23231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46129: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
I20260928 08:43:00.963908 23231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } attempt: 1
I20260928 08:43:00.965138 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.964690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:35924)
0928 08:43:00.964882 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.964978 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 08:43:00.966128 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.965843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:35938)
0928 08:43:00.965979 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.966041 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 08:43:00.966416 23231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46129: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
I20260928 08:43:00.985754 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } attempt: 3
I20260928 08:43:00.987013 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.986445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:35944)
0928 08:43:00.986695 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.986805 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 08:43:00.988155 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:00.987672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:35956)
0928 08:43:00.987933 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:00.988024 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260928 08:43:00.988384 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46129: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
I20260928 08:43:01.034577 23231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } attempt: 2
I20260928 08:43:01.035778 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.035319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:35958)
0928 08:43:01.035536 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:01.035619 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 08:43:01.036777 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.036444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:35962)
0928 08:43:01.036605 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:01.036671 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 08:43:01.037012 23231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46129: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
W20260928 08:43:01.121454 23313 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:43:01.121775 23313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:43:01.121838 23313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:43:01.121913 23313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:43:01.129366 23313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:43:01.129460 23313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:43:01.129519 23313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:43:01.129562 23313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:43:01.129614 23313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:43:01.131624 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } attempt: 4
I20260928 08:43:01.132884 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.132303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:35964)
0928 08:43:01.132562 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:01.132667 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 08:43:01.133880 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.133515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:35974)
0928 08:43:01.133662 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:01.133745 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 08:43:01.134091 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46129: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
I20260928 08:43:01.140540 23313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46241,127.0.97.61:42017,127.0.97.60:46129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:43:01.141789 23313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:43:01.143214 23231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } attempt: 3
I20260928 08:43:01.143299 23313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:43:01.144232 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.143806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:35984)
0928 08:43:01.143986 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:01.144082 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 08:43:01.145221 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.144876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:35996)
0928 08:43:01.145070 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:01.145130 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 08:43:01.145471 23231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46129: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46129: connect: Connection refused (error 111)
W20260928 08:43:01.155359 23318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:01.155330 23319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:01.155391 23321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:43:01.156404 23313 server_base.cc:1034] running on GCE node
I20260928 08:43:01.157230 23313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:43:01.158676 23313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:43:01.159914 23313 hybrid_clock.cc:648] HybridClock initialized: now 1790584981159856 us; error 45 us; skew 500 ppm
I20260928 08:43:01.160248 23313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:43:01 dist-test-slave-w6c0 krb5kdc[23151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:43:01.163467 23313 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260928 08:43:01.164942 23313 webserver.cc:460] Webserver started at http://127.0.97.60:44107/ using document root <none> and password file <none>
I20260928 08:43:01.165657 23313 fs_manager.cc:359] Metadata directory not provided
I20260928 08:43:01.165786 23313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:43:01.166101 23313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:43:01.169075 23313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-2/data/instance:
uuid: "5dafa71e28a94aec80113e00d771c058"
format_stamp: "Formatted at 2026-09-28 08:43:01 on dist-test-slave-w6c0"
server_key: "b3f2bed53589a70ab2528c4daf60fa6e"
server_key_iv: "8d9ce89ba9cb0d9dd09c4224a428956b"
server_key_version: "encryptionkey@0"
I20260928 08:43:01.169859 23313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-2/wal/instance:
uuid: "5dafa71e28a94aec80113e00d771c058"
format_stamp: "Formatted at 2026-09-28 08:43:01 on dist-test-slave-w6c0"
server_key: "b3f2bed53589a70ab2528c4daf60fa6e"
server_key_iv: "8d9ce89ba9cb0d9dd09c4224a428956b"
server_key_version: "encryptionkey@0"
I20260928 08:43:01.175035 23313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 08:43:01.178417 23328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:01.179797 23313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:43:01.179987 23313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-2/data,/tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-2/wal
uuid: "5dafa71e28a94aec80113e00d771c058"
format_stamp: "Formatted at 2026-09-28 08:43:01 on dist-test-slave-w6c0"
server_key: "b3f2bed53589a70ab2528c4daf60fa6e"
server_key_iv: "8d9ce89ba9cb0d9dd09c4224a428956b"
server_key_version: "encryptionkey@0"
I20260928 08:43:01.180207 23313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:43:01.202414 23313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:43:01.206403 23313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:43:01.206754 23313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:43:01.225430 23313 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46129
I20260928 08:43:01.225447 23379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46129 every 8 connection(s)
I20260928 08:43:01.227003 23313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-2/data/info.pb
I20260928 08:43:01.230212 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task38vn8d/build/asan/bin/kudu as pid 23313
I20260928 08:43:01.230406 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task38vn8d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584618551208-388-0/minicluster-data/master-2/wal/instance
I20260928 08:43:01.230999 388 external_mini_cluster.cc:1428] Setting key 99d894ff1fa38d209878a667854ad044
I20260928 08:43:01.231195 23380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:43:01.239696 23380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:43:01 dist-test-slave-w6c0 krb5kdc[23151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 28 08:43:01 dist-test-slave-w6c0 krb5kdc[23151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:43:01.254565 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.232797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36008 (local address 127.0.97.60:46129)
0928 08:43:01.233531 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:01.233542 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:01.233581 (+ 39us) server_negotiation.cc:407] Connection header received
0928 08:43:01.235851 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:01.235876 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:01.236287 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:01.236707 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:01.238158 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.239462 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.240756 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.241431 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.241624 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:01.245849 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:01.245859 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:01.245872 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:01.245915 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:01.249704 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.250576 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.250584 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.250593 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.250806 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.251217 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.251225 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.251229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.251965 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:01.252258 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:01.252771 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 08:43:01.253193 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":319,"threads_started":1}
I20260928 08:43:01.260623 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.241998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46241 (local address 127.0.0.1:33492)
0928 08:43:01.244632 (+ 2634us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:01.244670 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:01.244849 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:01.245258 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:01.245265 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:01.246028 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:01.246392 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:01.246405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.247610 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.247614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:01.249370 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:01.249387 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.250309 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.250313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:01.250432 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:01.251365 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:01.251392 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:01.254072 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:01.257765 (+ 3693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:01.257776 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:01.257783 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:01.258320 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:01.258821 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:01.258825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:01.258827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:01.258991 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:01.259547 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:01.259556 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:01.259764 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 08:43:01.259990 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 08:43:01.260319 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2447,"thread_start_us":363,"threads_started":1}
I20260928 08:43:01.260664 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.242213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33492 (local address 127.0.97.62:46241)
0928 08:43:01.242430 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:01.242446 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:01.244771 (+ 2325us) server_negotiation.cc:407] Connection header received
0928 08:43:01.244981 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:01.244987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:01.245088 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:01.245203 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:01.246599 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.247438 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.249549 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.250135 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.250272 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:01.254249 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:01.254255 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:01.254258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:01.254291 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:01.257604 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.258517 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.258523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.258527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.258691 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.259118 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.259123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.259125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.259406 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:01.259509 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:01.260061 (+ 552us) server_negotiation.cc:299] Negotiation successful
0928 08:43:01.260234 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 08:43:01.262106 23380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:43:01 dist-test-slave-w6c0 krb5kdc[23151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 28 08:43:01 dist-test-slave-w6c0 krb5kdc[23151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 08:43:01.273844 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.258086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33498 (local address 127.0.97.62:46241)
0928 08:43:01.258730 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:01.258737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:01.258755 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:43:01.258804 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:01.258808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:01.258871 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:01.258953 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:01.260392 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.261598 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.263138 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.263547 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.263684 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:01.267695 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:01.267702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:01.267707 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:01.267747 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:01.270950 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.271784 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.271788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.271790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.271907 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.272325 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.272328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.272330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.272601 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:01.272816 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:01.273197 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 08:43:01.273358 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":247,"threads_started":1}
I20260928 08:43:01.276467 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.262872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42017 (local address 127.0.0.1:50386)
0928 08:43:01.263037 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:01.263055 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:01.263142 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:01.263492 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:01.263496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:01.263852 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:01.263985 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:01.263993 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.264953 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.264958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:01.265806 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:01.265817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.266800 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.266807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:01.266979 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:01.267514 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:01.267531 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:01.270063 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:01.273741 (+ 3678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:01.273746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:01.273749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:01.274306 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:01.274773 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:01.274779 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:01.274782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:01.274927 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:01.275461 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:01.275467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:01.275637 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 08:43:01.275808 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 08:43:01.276104 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 08:43:01.276526 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.262878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50386 (local address 127.0.97.61:42017)
0928 08:43:01.263066 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:01.263074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:01.263110 (+ 36us) server_negotiation.cc:407] Connection header received
0928 08:43:01.263277 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:01.263283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:01.263359 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:01.263470 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:01.264132 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.264801 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.265957 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.266632 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.266762 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:01.270254 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:01.270259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:01.270262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:01.270295 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:01.273576 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.274425 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.274431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.274434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.274659 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.275039 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.275045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.275048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.275329 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:01.275646 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:01.275926 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 08:43:01.276134 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 08:43:01.277637 23380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:43:01 dist-test-slave-w6c0 krb5kdc[23151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 28 08:43:01 dist-test-slave-w6c0 krb5kdc[23151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 08:43:01.291250 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.278505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36018 (local address 127.0.97.60:46129)
0928 08:43:01.278679 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:01.278684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:01.278703 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:43:01.278749 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:01.278753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:01.278825 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:01.278930 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:01.279790 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.280697 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.281981 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.282576 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.282715 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:01.285900 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:01.285907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:01.285912 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:01.285949 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:01.288762 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.289388 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.289394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.289404 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.289507 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.289848 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.289856 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.289859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.290138 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:01.290285 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:01.290642 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 08:43:01.290863 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:43:01.291368 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.278283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:36018)
0928 08:43:01.278441 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:01.278459 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:01.278571 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:01.278978 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:01.278982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:01.279485 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:01.279659 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:01.279670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.280821 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.280824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:01.281841 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:01.281855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.282759 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.282766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:01.282983 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:01.283544 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:01.283560 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:01.285725 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:01.288907 (+ 3182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:01.288912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:01.288916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:01.289270 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:01.289618 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:01.289623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:01.289627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:01.289750 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:01.290249 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:01.290253 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:01.290394 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 08:43:01.290568 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 08:43:01.291016 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":144}
I20260928 08:43:01.292327 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.276193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50400 (local address 127.0.97.61:42017)
0928 08:43:01.276834 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:01.276839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:01.277070 (+ 231us) server_negotiation.cc:407] Connection header received
0928 08:43:01.277248 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:01.277254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:01.277329 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:01.277432 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:01.278991 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.280027 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.281449 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.282085 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.282172 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:01.286767 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:01.286772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:01.286775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:01.286805 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:01.289223 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.289888 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.289894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.289899 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.290032 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.290423 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.290434 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.290438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.290759 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:01.291039 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:01.291720 (+ 681us) server_negotiation.cc:299] Negotiation successful
0928 08:43:01.291987 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":1076,"threads_started":1}
I20260928 08:43:01.296636 23380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058: Bootstrap starting.
I20260928 08:43:01.296856 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:43:01.300436 23380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058: Neither blocks nor log segments found. Creating new log.
I20260928 08:43:01.301483 23380 log.cc:824] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058: Log is configured to *not* fsync() on all Append() calls
I20260928 08:43:01.299896 23206 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cf2ebd8da3484365bfb291aaf6dd701e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33498
I20260928 08:43:01.304304 23380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058: No bootstrap required, opened a new log
I20260928 08:43:01.311544 23380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } } peers { permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } } peers { permanent_uuid: "5dafa71e28a94aec80113e00d771c058" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } }
I20260928 08:43:01.311879 23380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dafa71e28a94aec80113e00d771c058, State: Initialized, Role: FOLLOWER
I20260928 08:43:01.312320 23380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } } peers { permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } } peers { permanent_uuid: "5dafa71e28a94aec80113e00d771c058" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } }
I20260928 08:43:01.313241 23388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } } peers { permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } } peers { permanent_uuid: "5dafa71e28a94aec80113e00d771c058" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } } }
I20260928 08:43:01.313614 23388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:43:01.314344 23380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:43:01.322690 23231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } attempt: 4
W20260928 08:43:01.323943 23399 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:43:01.324064 23399 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:43:01.321715 23313 master_runner.cc:186] Error getting master registration for 127.0.97.62:46241: OK, instance_id { permanent_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" instance_seqno: 1790584980181753 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
Sep 28 08:43:01 dist-test-slave-w6c0 krb5kdc[23151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 08:43:01.333318 23313 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 08:43:01.340446 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.323996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36030 (local address 127.0.97.60:46129)
0928 08:43:01.324204 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:01.324210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:01.324237 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:43:01.324291 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:01.324296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:01.324375 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:01.324470 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:01.325601 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.326943 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.328804 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.329602 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.329725 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:01.334155 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:01.334162 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:01.334168 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:01.334212 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:01.337311 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.338175 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.338183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.338188 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.338347 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.338823 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.338828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.338831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.339257 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:01.339410 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:01.339842 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 08:43:01.340065 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:43:01.340458 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.323297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:36030)
0928 08:43:01.323638 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:01.323658 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:01.323771 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:01.324663 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:01.324668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:01.325214 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:01.325427 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:01.325440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.327143 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.327151 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:01.328622 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:01.328637 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.330002 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.330010 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:01.330216 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:01.330931 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:01.330957 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:01.333924 (+ 2967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:01.337470 (+ 3546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:01.337475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:01.337481 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:01.338036 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:01.338473 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:01.338479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:01.338483 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:01.338707 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:01.339390 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:01.339396 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:01.339562 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 08:43:01.339723 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 08:43:01.340052 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214}
I20260928 08:43:01.345499 23231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e: Bootstrap starting.
I20260928 08:43:01.348994 23231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e: Neither blocks nor log segments found. Creating new log.
I20260928 08:43:01.350337 23231 log.cc:824] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:43:01.353724 23231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e: No bootstrap required, opened a new log
I20260928 08:43:01.358880 23231 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } } peers { permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } } peers { permanent_uuid: "5dafa71e28a94aec80113e00d771c058" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } }
I20260928 08:43:01.359268 23231 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2ebd8da3484365bfb291aaf6dd701e, State: Initialized, Role: FOLLOWER
I20260928 08:43:01.359822 23231 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } } peers { permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } } peers { permanent_uuid: "5dafa71e28a94aec80113e00d771c058" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } }
I20260928 08:43:01.360757 23402 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } } peers { permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } } peers { permanent_uuid: "5dafa71e28a94aec80113e00d771c058" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } } }
I20260928 08:43:01.361124 23402 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:43:01.361904 23231 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:43:01.362046 23206 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 08:43:01.362253 23206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:43:01.366081 23206 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } } peers { permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } } peers { permanent_uuid: "5dafa71e28a94aec80113e00d771c058" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } }
I20260928 08:43:01.369768 23355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5dafa71e28a94aec80113e00d771c058"
I20260928 08:43:01.370287 23355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:43:01.373961 23355 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf2ebd8da3484365bfb291aaf6dd701e in term 1.
I20260928 08:43:01.374604 23182 leader_election.cc:302] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dafa71e28a94aec80113e00d771c058, cf2ebd8da3484365bfb291aaf6dd701e; no voters:
I20260928 08:43:01.375139 23402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:43:01.378396 23206 leader_election.cc:288] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [CANDIDATE]: Term 1 election: Requested vote from peers 189780c28f364bf3b705ad1e229e72c9 (127.0.97.61:42017), 5dafa71e28a94aec80113e00d771c058 (127.0.97.60:46129)
I20260928 08:43:01.385929 23402 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [term 1 LEADER]: Becoming Leader. State: Replica: cf2ebd8da3484365bfb291aaf6dd701e, State: Running, Role: LEADER
I20260928 08:43:01.386636 23402 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } } peers { permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } } peers { permanent_uuid: "5dafa71e28a94aec80113e00d771c058" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } }
I20260928 08:43:01.372534 23164 master_runner.cc:186] Error getting master registration for 127.0.97.61:42017: OK, instance_id { permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" instance_seqno: 1790584980706551 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:43:01.391013 23164 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 08:43:01.393666 23413 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:43:01.393791 23413 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:43:01.411899 23415 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf2ebd8da3484365bfb291aaf6dd701e. Latest consensus state: current_term: 1 leader_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } } peers { permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } } peers { permanent_uuid: "5dafa71e28a94aec80113e00d771c058" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } } }
I20260928 08:43:01.412389 23415 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [sys.catalog]: This master's current role is: LEADER
W20260928 08:43:01.420985 23185 tablet.cc:2369] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:43:01.423574 23423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:43:01.425786 23423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:43:01.433282 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } attempt: 5
I20260928 08:43:01.441990 23355 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058 [term 1 FOLLOWER]: Refusing update from remote peer cf2ebd8da3484365bfb291aaf6dd701e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:43:01.443079 23415 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dafa71e28a94aec80113e00d771c058" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:43:01.455370 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.399744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33514 (local address 127.0.97.62:46241)
0928 08:43:01.400317 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:01.400327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:01.416822 (+ 16495us) server_negotiation.cc:407] Connection header received
0928 08:43:01.417598 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:01.417604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:01.417694 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:01.417778 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:01.422728 (+ 4950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.423847 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.426106 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.427016 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.427127 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:01.438804 (+ 11677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:01.438811 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:01.438817 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:01.438861 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:01.445169 (+ 6308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.446928 (+ 1759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.446934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.446938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.447469 (+ 531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.448648 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.448654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.448657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.449928 (+ 1271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:01.452909 (+ 2981us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:01.453431 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 08:43:01.454210 (+ 779us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374}
I20260928 08:43:01.457262 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.400560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.0.97.61:42017)
0928 08:43:01.400790 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:01.400799 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:01.415358 (+ 14559us) server_negotiation.cc:407] Connection header received
0928 08:43:01.415925 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:01.415934 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:01.416054 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:01.416176 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:01.420748 (+ 4572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.421939 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.427249 (+ 5310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.427966 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.428082 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:01.435962 (+ 7880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:01.435970 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:01.435975 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:01.436017 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:01.443115 (+ 7098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.446874 (+ 3759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.446881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.446885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.447526 (+ 641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.448711 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.448716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.448720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.449980 (+ 1260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:01.452824 (+ 2844us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:01.453252 (+ 428us) server_negotiation.cc:299] Negotiation successful
0928 08:43:01.454148 (+ 896us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260928 08:43:01.459013 23334 tablet.cc:2369] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:43:01.467904 23388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf2ebd8da3484365bfb291aaf6dd701e. Latest consensus state: current_term: 1 leader_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } } peers { permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } } peers { permanent_uuid: "5dafa71e28a94aec80113e00d771c058" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } } }
I20260928 08:43:01.468259 23388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:43:01.472548 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.413641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36034 (local address 127.0.97.60:46129)
0928 08:43:01.413889 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:01.413898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:01.425481 (+ 11583us) server_negotiation.cc:407] Connection header received
0928 08:43:01.426227 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:01.426233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:01.426314 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:01.426391 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:01.431573 (+ 5182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.432752 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.460480 (+ 27728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.461299 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.461410 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:01.464955 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:01.464962 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:01.464970 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:01.465011 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:01.469243 (+ 4232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.470102 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.470109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.470113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.470272 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.470740 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.470745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.470749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.471087 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:01.471337 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:01.471903 (+ 566us) server_negotiation.cc:299] Negotiation successful
0928 08:43:01.472137 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 08:43:01 dist-test-slave-w6c0 krb5kdc[23151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 08:43:01.482242 23425 mvcc.cc:205] Tried to move back new op lower bound from 7334236083973820416 to 7334236083765936128. Current Snapshot: MvccSnapshot[applied={T|T < 7334236083973820416}]
I20260928 08:43:01.494493 23415 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } } peers { permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } } peers { permanent_uuid: "5dafa71e28a94aec80113e00d771c058" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } } }
I20260928 08:43:01.494848 23415 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [sys.catalog]: This master's current role is: LEADER
I20260928 08:43:01.496718 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.434026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46129 (local address 127.0.0.1:36044)
0928 08:43:01.434471 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:01.434495 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:01.434640 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:01.459843 (+ 25203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:01.459850 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:01.460351 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:01.460551 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:01.460563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.469032 (+ 8469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.469039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:01.470567 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:01.470586 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.472538 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.472545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:01.472782 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:01.473413 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:01.473451 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:01.478196 (+ 4745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:01.492660 (+ 14464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:01.492674 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:01.492679 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:01.493327 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:01.494068 (+ 741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:01.494075 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:01.494078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:01.494253 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:01.495687 (+ 1434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:01.495693 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:01.495865 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 08:43:01.496013 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 08:43:01.496319 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284}
I20260928 08:43:01.499119 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.435062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.0.97.60:46129)
0928 08:43:01.457431 (+ 22369us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:01.457438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:01.457574 (+ 136us) server_negotiation.cc:407] Connection header received
0928 08:43:01.459027 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:01.459036 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:01.459425 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:01.460872 (+ 1447us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:01.462383 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.468861 (+ 6478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.470817 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.472247 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.473153 (+ 906us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:01.478566 (+ 5413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:01.478577 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:01.478582 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:01.478619 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:01.492338 (+ 13719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.493614 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.493619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.493623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.493877 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:01.494459 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:01.494463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:01.494468 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:01.495467 (+ 999us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:01.497748 (+ 2281us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:01.497989 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 08:43:01.498393 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21533,"thread_start_us":20565,"threads_started":1}
I20260928 08:43:01.499549 23423 catalog_manager.cc:1347] Generated new cluster ID: 815468c4b02d44d2abed15485ffc54f5
I20260928 08:43:01.499655 23423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:43:01.511999 23305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9: Bootstrap starting.
I20260928 08:43:01.516721 23305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9: Neither blocks nor log segments found. Creating new log.
I20260928 08:43:01.520612 23388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } } peers { permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } } peers { permanent_uuid: "5dafa71e28a94aec80113e00d771c058" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } } }
I20260928 08:43:01.520881 23388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:43:01.518409 23305 log.cc:824] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9: Log is configured to *not* fsync() on all Append() calls
I20260928 08:43:01.534618 23305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9: No bootstrap required, opened a new log
I20260928 08:43:01.551528 23423 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:43:01.553589 23423 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:43:01.553249 23305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } } peers { permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } } peers { permanent_uuid: "5dafa71e28a94aec80113e00d771c058" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } }
I20260928 08:43:01.553969 23305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 189780c28f364bf3b705ad1e229e72c9, State: Initialized, Role: FOLLOWER
I20260928 08:43:01.554939 23305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } } peers { permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } } peers { permanent_uuid: "5dafa71e28a94aec80113e00d771c058" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } }
I20260928 08:43:01.567060 23432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } } peers { permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } } peers { permanent_uuid: "5dafa71e28a94aec80113e00d771c058" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } } }
I20260928 08:43:01.567515 23432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:43:01.571108 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "189780c28f364bf3b705ad1e229e72c9"
I20260928 08:43:01.571584 23280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:43:01.575030 23280 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf2ebd8da3484365bfb291aaf6dd701e in term 1.
I20260928 08:43:01.575847 23279 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9 [term 1 FOLLOWER]: Refusing update from remote peer cf2ebd8da3484365bfb291aaf6dd701e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:43:01.577095 23415 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [LEADER]: Connected to new peer: Peer: permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:43:01.582664 23423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e: Generated new TSK 0
I20260928 08:43:01.584508 23423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:43:01.586549 23305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:43:01.653276 23432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf2ebd8da3484365bfb291aaf6dd701e. Latest consensus state: current_term: 1 leader_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } } peers { permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } } peers { permanent_uuid: "5dafa71e28a94aec80113e00d771c058" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } } }
I20260928 08:43:01.653287 23441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } } peers { permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } } peers { permanent_uuid: "5dafa71e28a94aec80113e00d771c058" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } } }
I20260928 08:43:01.653985 23432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:43:01.654047 23441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:43:01.668159 23450 catalog_manager.cc:1259] Loaded cluster ID: 815468c4b02d44d2abed15485ffc54f5
I20260928 08:43:01.668327 23450 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9: loading cluster ID for follower catalog manager: success
I20260928 08:43:01.671929 23450 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9: acquiring CA information for follower catalog manager: success
I20260928 08:43:01.674415 23450 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 189780c28f364bf3b705ad1e229e72c9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 08:43:01.762820 23310 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:43:01.744960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50416 (local address 127.0.97.61:42017)
0928 08:43:01.745200 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:01.745207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:01.760174 (+ 14967us) server_negotiation.cc:407] Connection header received
0928 08:43:01.761220 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:01.761227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:01.761312 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:01.761397 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:01.762569 (+ 1172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50416: BlockingRecv error: recv got EOF from 127.0.0.1:50416 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 08:43:01.763571 23385 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:43:01.738685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33528 (local address 127.0.97.62:46241)
0928 08:43:01.738891 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:01.738898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:01.754151 (+ 15253us) server_negotiation.cc:407] Connection header received
0928 08:43:01.754374 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:01.754381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:01.754537 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:01.754667 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:01.763274 (+ 8607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33528: BlockingRecv error: recv got EOF from 127.0.0.1:33528 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 08:43:01.764693 23426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:43:01.754897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36048 (local address 127.0.97.60:46129)
0928 08:43:01.755082 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:01.755093 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:01.760825 (+ 5732us) server_negotiation.cc:407] Connection header received
0928 08:43:01.761033 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:01.761040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:01.761152 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:01.761268 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:01.764459 (+ 3191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36048: BlockingRecv error: recv got EOF from 127.0.0.1:36048 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 08:43:01.813287 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.797398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50422 (local address 127.0.97.61:42017)
0928 08:43:01.797602 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:01.797609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:01.804999 (+ 7390us) server_negotiation.cc:407] Connection header received
0928 08:43:01.805584 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:01.805593 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:01.805812 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:01.805980 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:43:01.806702 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.807913 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.810019 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.810739 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.810844 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:01.811656 (+ 812us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:43:01.812397 (+ 741us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:43:01.812527 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:01.812806 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 08:43:01.812911 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 08:43:01.857611 23426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:43:01.847490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36054 (local address 127.0.97.60:46129)
0928 08:43:01.847684 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:01.847690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:01.855457 (+ 7767us) server_negotiation.cc:407] Connection header received
0928 08:43:01.856008 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:01.856016 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:01.856151 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:01.856236 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:01.857308 (+ 1072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36054: BlockingRecv error: recv got EOF from 127.0.0.1:36054 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:43:01.903703 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.888890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.0.97.61:42017)
0928 08:43:01.889133 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:01.889141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:01.896595 (+ 7454us) server_negotiation.cc:407] Connection header received
0928 08:43:01.897147 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:01.897155 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:01.897262 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:01.897335 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:43:01.898017 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.899239 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.901283 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.901952 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.902048 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:01.902736 (+ 688us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:43:01.902924 (+ 188us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:43:01.902997 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:01.903263 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 08:43:01.903369 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 08:43:01.947883 23426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:43:01.937775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36064 (local address 127.0.97.60:46129)
0928 08:43:01.937981 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:01.937988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:01.945595 (+ 7607us) server_negotiation.cc:407] Connection header received
0928 08:43:01.946192 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:01.946201 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:01.946330 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:01.946453 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:01.947638 (+ 1185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36064: BlockingRecv error: recv got EOF from 127.0.0.1:36064 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 08:43:01.964145 23402 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2ebd8da3484365bfb291aaf6dd701e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46241 } } peers { permanent_uuid: "189780c28f364bf3b705ad1e229e72c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42017 } } peers { permanent_uuid: "5dafa71e28a94aec80113e00d771c058" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46129 } } }
I20260928 08:43:01.964514 23402 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2ebd8da3484365bfb291aaf6dd701e [sys.catalog]: This master's current role is: LEADER
I20260928 08:43:01.996702 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:01.986749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50436 (local address 127.0.97.61:42017)
0928 08:43:01.987017 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:01.987026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:01.990349 (+ 3323us) server_negotiation.cc:407] Connection header received
0928 08:43:01.990543 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:01.990552 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:01.990656 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:01.990771 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:43:01.991275 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.992498 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.994351 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:01.995064 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:01.995207 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:01.995759 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:43:01.995923 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:43:01.996026 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:01.996208 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 08:43:01.996330 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260928 08:43:02.008829 23426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:43:02.004391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.0.97.60:46129)
0928 08:43:02.004598 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:02.004605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:02.007141 (+ 2536us) server_negotiation.cc:407] Connection header received
0928 08:43:02.007370 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:02.007376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:02.007495 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:02.007615 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:02.008564 (+ 949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36074: BlockingRecv error: recv got EOF from 127.0.0.1:36074 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 08:43:02.023229 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:02.016084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.0.97.61:42017)
0928 08:43:02.016340 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:02.016348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:02.016887 (+ 539us) server_negotiation.cc:407] Connection header received
0928 08:43:02.017105 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:02.017111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:02.017224 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:02.017333 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:43:02.017859 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.019071 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.021025 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.021733 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.021870 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:02.022341 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:43:02.022496 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:43:02.022648 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:02.022825 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 08:43:02.022920 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
W20260928 08:43:02.032337 23426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:43:02.030566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36086 (local address 127.0.97.60:46129)
0928 08:43:02.030807 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:02.030814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:02.030835 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:43:02.030912 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:02.030917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:02.031025 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:02.031140 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:02.032046 (+ 906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36086: BlockingRecv error: recv got EOF from 127.0.0.1:36086 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
I20260928 08:43:02.052372 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:02.044439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50458 (local address 127.0.97.61:42017)
0928 08:43:02.044629 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:02.044635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:02.045197 (+ 562us) server_negotiation.cc:407] Connection header received
0928 08:43:02.045653 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:02.045660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:02.045784 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:02.045867 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:43:02.046801 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.048013 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.050049 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.050810 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.050960 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:02.051465 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:43:02.051627 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:43:02.051737 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:02.051905 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 08:43:02.051997 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 08:43:02.066251 23426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:43:02.064129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36096 (local address 127.0.97.60:46129)
0928 08:43:02.064350 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:02.064357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:02.064380 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:43:02.064707 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:02.064714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:02.064824 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:02.064942 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:02.065941 (+ 999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36096: BlockingRecv error: recv got EOF from 127.0.0.1:36096 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 08:43:02.094408 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:02.086914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.0.97.61:42017)
0928 08:43:02.087118 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:02.087125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:02.087720 (+ 595us) server_negotiation.cc:407] Connection header received
0928 08:43:02.087952 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:02.087958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:02.088070 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:02.088175 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:43:02.088667 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.090028 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.092087 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.092830 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.092996 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:02.093527 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:43:02.093702 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:43:02.093823 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:02.093987 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 08:43:02.094084 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 08:43:02.107012 23426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:43:02.104684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36106 (local address 127.0.97.60:46129)
0928 08:43:02.104873 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:02.104880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:02.105438 (+ 558us) server_negotiation.cc:407] Connection header received
0928 08:43:02.105629 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:02.105634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:02.105745 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:02.105854 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:02.106762 (+ 908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36106: BlockingRecv error: recv got EOF from 127.0.0.1:36106 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:43:02.149057 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:02.142845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.0.97.61:42017)
0928 08:43:02.143188 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:02.143194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:02.143437 (+ 243us) server_negotiation.cc:407] Connection header received
0928 08:43:02.143652 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:02.143656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:02.143726 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:02.143856 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:43:02.144297 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.145056 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.147035 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.147504 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.147615 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:02.148156 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:43:02.148267 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:43:02.148340 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:02.148559 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 08:43:02.148631 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
W20260928 08:43:02.157801 23426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:43:02.155912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36120 (local address 127.0.97.60:46129)
0928 08:43:02.156120 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:02.156126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:02.156360 (+ 234us) server_negotiation.cc:407] Connection header received
0928 08:43:02.156526 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:02.156531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:02.156617 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:02.156723 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:02.157521 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36120: BlockingRecv error: recv got EOF from 127.0.0.1:36120 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 08:43:02.230963 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:02.225216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50490 (local address 127.0.97.61:42017)
0928 08:43:02.225407 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:02.225414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:02.225729 (+ 315us) server_negotiation.cc:407] Connection header received
0928 08:43:02.225908 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:02.225913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:02.226019 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:02.226137 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:43:02.226547 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.227649 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.229096 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.229706 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.229842 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:02.230196 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:43:02.230331 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:43:02.230419 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:02.230585 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 08:43:02.230673 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 08:43:02.239037 23426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:43:02.237036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36128 (local address 127.0.97.60:46129)
0928 08:43:02.237268 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:02.237275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:02.237543 (+ 268us) server_negotiation.cc:407] Connection header received
0928 08:43:02.237708 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:02.237712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:02.237795 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:02.237888 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:02.238814 (+ 926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36128: BlockingRecv error: recv got EOF from 127.0.0.1:36128 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 08:43:02.326046 23399 catalog_manager.cc:1259] Loaded cluster ID: 815468c4b02d44d2abed15485ffc54f5
I20260928 08:43:02.326150 23399 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058: loading cluster ID for follower catalog manager: success
I20260928 08:43:02.328760 23399 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058: acquiring CA information for follower catalog manager: success
I20260928 08:43:02.330574 23399 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5dafa71e28a94aec80113e00d771c058: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:43:02.376537 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:02.370628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.0.97.61:42017)
0928 08:43:02.370891 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:02.370898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:02.371124 (+ 226us) server_negotiation.cc:407] Connection header received
0928 08:43:02.371377 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:02.371382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:02.371463 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:02.371565 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:43:02.372002 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.372729 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.374701 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.375137 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.375286 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:02.375772 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:43:02.375885 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:43:02.375992 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:02.376157 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 08:43:02.376243 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 08:43:02.388895 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:02.383034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36130 (local address 127.0.97.60:46129)
0928 08:43:02.383205 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:02.383209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:02.383564 (+ 355us) server_negotiation.cc:407] Connection header received
0928 08:43:02.383728 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:02.383732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:02.383912 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:02.384052 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:43:02.384504 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.385283 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.386632 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.387082 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.387254 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:02.387745 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:43:02.388282 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:43:02.388461 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:02.388511 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 08:43:02.388597 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:43:02.404038 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:02.394949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50506 (local address 127.0.97.61:42017)
0928 08:43:02.395188 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:02.395211 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:02.395735 (+ 524us) server_negotiation.cc:407] Connection header received
0928 08:43:02.396015 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:02.396022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:02.396116 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:02.396185 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:43:02.397144 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.398653 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.401513 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.402305 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.402378 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:02.403209 (+ 831us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:43:02.403361 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:43:02.403466 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:02.403651 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 08:43:02.403751 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 08:43:02.408061 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:02.395883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36134 (local address 127.0.97.60:46129)
0928 08:43:02.396048 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:02.396053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:02.402470 (+ 6417us) server_negotiation.cc:407] Connection header received
0928 08:43:02.402734 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:02.402739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:02.402812 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:02.402916 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:43:02.403598 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.404443 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.406253 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.406739 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.406836 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:02.407311 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:43:02.407437 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:43:02.407504 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:02.407660 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 08:43:02.407726 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:43:02.408704 23555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:02.394847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33538 (local address 127.0.97.62:46241)
0928 08:43:02.401385 (+ 6538us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:02.401394 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:02.401756 (+ 362us) server_negotiation.cc:407] Connection header received
0928 08:43:02.402003 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:02.402011 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:02.402274 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:02.402465 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:43:02.403225 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.404394 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.406308 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:02.406819 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:02.406952 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:02.407432 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:43:02.407921 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:43:02.408001 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:02.408171 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 08:43:02.408268 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6397,"thread_start_us":6282,"threads_started":1}
I20260928 08:43:02.413640 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 23164
I20260928 08:43:02.414688 23226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:43:02.528916 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 23164
I20260928 08:43:02.550571 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 23238
I20260928 08:43:02.551697 23300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:43:02.666117 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 23238
I20260928 08:43:02.687686 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task38vn8d/build/asan/bin/kudu pid 23313
I20260928 08:43:02.688784 23375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:43:02.796823 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task38vn8d/build/asan/bin/kudu with pid 23313
2026-09-28T08:43:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4401 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9381 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (364207 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST